Model training method, multi-modal content generation method and device

Abstract

Embodiments of the present specification provide a model training method, a multi-modal content generation method and apparatus, wherein the model training method comprises: obtaining a sample prompt word, a target multi-modal generation model and a pre-training inference model; using the pre-training inference model, performing multiple inferences based on the sample prompt word to obtain multiple sample inference tracks; using the target multi-modal generation model, generating sample candidate images corresponding to the multiple sample inference tracks based on the multiple sample inference tracks; performing advantage evaluation on each sample candidate image to obtain a relative advantage value of each sample candidate image; based on the relative advantage value, selecting a positive advantage sample inference track from the multiple sample inference tracks; and based on the positive advantage sample inference track, performing supervised fine-tuning training on the pre-training inference model to obtain a target inference model. The pre-training inference model can naturally learn the generation capability of the target multi-modal generation model from the training process and give reasonable inferences on the sample prompt word.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large model technology, and in particular to a model training method and a multimodal content generation method. Background Technology

[0002] In recent years, text-to-image (Text to Image) to Breakthroughs have been achieved in image generation models. Generative architectures, exemplified by diffusion models, can synthesize high-quality images with accurate lighting, coherent spatial structure, and complex scene details. Meanwhile, these models are being widely applied in content creation, advertising design, virtual reality, and other scenarios, becoming a crucial technological foundation for the field of multimodal content generation.

[0003] While existing text-to-image generation models perform well on standard benchmarks, in real-world request scenarios, many prompts involve knowledge outside the model's training distribution, such as recently emerging characters and events, regional cultural symbols, niche fonts, historical artifacts, and specific images. Faced with such requests, generators often produce beautifully rendered but completely erroneous images.

[0004] Search is a natural means of addressing this bottleneck; however, existing technologies lack a mechanism to determine when retrieval is necessary. For prompts where the generator already possesses accurate knowledge, performing a search can introduce concept erosion. Furthermore, for requests outside the generator's knowledge boundaries, directly copying retrieved reference images instead of creatively generating new ones can introduce a copy effect. Therefore, accurately identifying the generator's knowledge gaps and effectively utilizing search information while avoiding concept erosion and the copy effect is a pressing issue. Summary of the Invention

[0005] In view of the above, embodiments of this specification provide a model training method. One or more embodiments of this specification also relate to a content reasoning method, a multimodal content generation method, a reasoning model training method, an image generation method, a model training apparatus, a content reasoning apparatus, a multimodal content generation apparatus, an image generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising: Acquire sample prompts, target multimodal generation model, and pre-trained inference model; By using a pre-trained inference model, multiple inferences are performed based on sample prompts to obtain multiple sample inference trajectories. Using a target multimodal generative model, candidate images of samples corresponding to multiple sample inference trajectories are generated based on multiple sample inference trajectories; The relative advantage value of each candidate image is obtained by evaluating the advantage of each candidate image. Based on the relative advantage value, the positive advantage sample inference trajectory is selected from multiple sample inference trajectories; Based on the positive dominance sample inference trajectory, the pre-trained inference model is subjected to supervised fine-tuning training to obtain the target inference model.

[0007] According to a second aspect of the embodiments of this specification, a content reasoning method is provided, comprising: The initial content is input into the target inference model, and a search gating task is performed to obtain multiple candidate knowledge gaps corresponding to the initial content. The target knowledge gap corresponding to the initial content is then determined from the candidate knowledge gaps. The target reasoning model is used to perform a reference filtering task. Based on the target knowledge gap corresponding to the initial content, the image information of multiple candidate reference images corresponding to the initial content is retrieved, and the image information of the target reference image corresponding to the initial content is determined from the image information of the multiple candidate reference images corresponding to the initial content. The target reasoning model is used to perform a reference integration task. The initial content is adjusted based on the target knowledge gap corresponding to the initial content and the image information of the target reference image corresponding to the initial content to obtain the target reasoning content.

[0008] According to a third aspect of the embodiments of this specification, a multimodal content generation method is provided, comprising: Obtain the prompt words to be generated, the target inference model, and the target multimodal generation model, wherein the target inference model and the target multimodal generation model are obtained according to the model training method; Using a target reasoning model, reasoning is performed based on the prompt words to be generated to obtain reference prompt words; Using a target multimodal generation model, target multimodal content is generated based on reference prompts.

[0009] According to a fourth aspect of the embodiments of this specification, a method for training a reasoning model is provided, comprising: Acquire sample prompts, target image generation models, and pre-trained inference models; By using a pre-trained inference model, multiple inferences are performed based on sample prompts to obtain multiple sample inference trajectories. Using a target image generation model, candidate images of samples corresponding to multiple sample inference trajectories are generated based on multiple sample inference trajectories; The relative advantage value of each candidate image is obtained by evaluating the advantage of each candidate image. Based on the relative advantage value, the positive advantage sample inference trajectory is selected from multiple sample inference trajectories; Based on the positive dominance sample inference trajectory, the pre-trained inference model is subjected to supervised fine-tuning training to obtain the target inference model.

[0010] According to a fifth aspect of the embodiments of this specification, an image generation method is provided, comprising: Obtain the prompt words to be generated, the target inference model, and the target image generation model, wherein the target inference model and the target image generation model are obtained according to the inference model training method; Using a target reasoning model, reasoning is performed based on the prompt words to be generated to obtain reference prompt words; A target image is generated using a target image generation model based on reference prompts.

[0011] According to a sixth aspect of the embodiments of this specification, a model training apparatus is provided, comprising: The first acquisition module is configured to acquire sample prompt words, target multimodal generation model, and pre-trained inference model; The first reasoning module is configured to use a pre-trained reasoning model to perform multiple inferences based on sample prompts and obtain multiple sample reasoning trajectories. The first candidate image generation module is configured to use the target multimodal generation model to generate candidate images of multiple sample inference trajectories based on multiple sample inference trajectories; The first advantage assessment module is configured to assess the advantage of each candidate image and obtain the relative advantage value of each candidate image. The first screening module is configured to select positive advantage sample inference trajectories from multiple sample inference trajectories based on the relative advantage value. The first supervised fine-tuning module is configured to perform supervised fine-tuning training on the pre-trained inference model based on the inference trajectory of positive dominance samples to obtain the target inference model.

[0012] According to a seventh aspect of the embodiments of this specification, a reasoning model training apparatus is provided, comprising: The gating task module is configured to input initial content into the target inference model, perform a search gating task, obtain multiple candidate knowledge gaps corresponding to the initial content, and determine the target knowledge gap corresponding to the initial content from the candidate knowledge gaps. The reference filtering module is configured to perform a reference filtering task using the target inference model, retrieve image information of multiple candidate reference images corresponding to the initial content based on the target knowledge gap corresponding to the initial content, and determine the image information of the target reference image corresponding to the initial content from the image information of the multiple candidate reference images corresponding to the initial content. The reference integration module is configured to perform a reference integration task using the target inference model. It adjusts the initial content based on the target knowledge gap corresponding to the initial content and the image information of the target reference image corresponding to the initial content to obtain the target inference content.

[0013] According to an eighth aspect of the embodiments of this specification, a multimodal content generation apparatus is provided, comprising: The second acquisition module is configured to acquire the prompt words to be generated, the target inference model, and the target multimodal generation model, wherein the target inference model and the target multimodal generation model are obtained according to the model training method; The second reasoning module is configured to use the target reasoning model to reason based on the prompt words to be generated, and obtain reference prompt words. The multimodal generation module is configured to generate target multimodal content based on reference prompts using a target multimodal generation model.

[0014] According to a ninth aspect of the embodiments of this specification, a reasoning model training apparatus is provided, comprising: The third acquisition module is configured to acquire sample prompt words, target image generation model and pre-trained inference model; The third reasoning module is configured to use a pre-trained reasoning model to perform multiple inferences based on sample prompts and obtain multiple sample reasoning trajectories. The second candidate image generation module is configured to use the target image generation model to generate candidate images of multiple sample inference trajectories based on multiple sample inference trajectories. The second advantage evaluation module is configured to evaluate the advantage of each candidate image and obtain the relative advantage value of each candidate image. The second filtering module is configured to filter out positive advantage sample inference trajectories from multiple sample inference trajectories based on the relative advantage value. The second supervised fine-tuning module is configured to perform supervised fine-tuning training on the pre-trained inference model based on the inference trajectory of positive dominance samples to obtain the target inference model.

[0015] According to a tenth aspect of the embodiments of this specification, an image generation apparatus is provided, comprising: The fourth acquisition module is configured to acquire the prompt words to be generated, the target inference model, and the target image generation model, wherein the target inference model and the target image generation model are obtained according to the inference model training method; The fourth reasoning module is configured to use the target reasoning model to reason based on the prompt words to be generated, and obtain reference prompt words; The image generation module is configured to generate a target image based on reference prompts using a target image generation model.

[0016] According to an eleventh aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of any of the above-mentioned model training method, multimodal content generation method, inference model training method, and image generation method.

[0017] According to a twelfth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of any of the above-described model training method, multimodal content generation method, inference model training method, and image generation method.

[0018] According to a thirteenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any one of the above-described model training method, multimodal content generation method, inference model training method, and image generation method.

[0019] The model training method provided in one or more embodiments of this specification includes: acquiring sample prompt words, a target multimodal generative model, and a pre-trained inference model; using the pre-trained inference model to perform multiple inferences based on the sample prompt words to obtain multiple sample inference trajectories; using the target multimodal generative model to generate sample candidate images corresponding to the multiple sample inference trajectories based on the multiple sample inference trajectories; evaluating the advantage of each sample candidate image to obtain the relative advantage value of each sample candidate image; selecting positive advantage sample inference trajectories from the multiple sample inference trajectories based on the relative advantage value; and performing supervised fine-tuning training on the pre-trained inference model based on the positive advantage sample inference trajectories to obtain the target inference model.

[0020] By acquiring sample prompts, a target multimodal generation model, and a pre-trained inference model, and using the pre-trained inference model to perform multiple inferences based on the sample prompts to obtain multiple sample inference trajectories, diverse sample inference trajectories are automatically generated without relying on manual annotation. The target multimodal generation model generates corresponding sample candidate images based on each sample inference trajectory, allowing the actual generation quality of each sample inference trajectory to be quantified in image form. By evaluating the advantages of each sample candidate image, a relative advantage value is obtained, enabling comparison of sample inference trajectories from different prompts on a uniform relative scale, thus accurately distinguishing which sample inference trajectories produce superior generation results. Based on this relative advantage value, positively dominant sample inference trajectories are selected from multiple sample inference trajectories, retaining those that can guide the generation of higher-quality images. The pre-trained inference model is then subjected to supervised fine-tuning training based on the positively dominant sample inference trajectories, allowing the pre-trained inference model to naturally learn the generation capabilities of the target multimodal generation model during training without explicit annotation, and to provide reasonable inferences for the sample prompts, achieving calibration between the target inference model's search strategy and the target multimodal generation model's capabilities. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a model training method provided in one embodiment of this specification; Figure 2 A flowchart illustrating the processing steps of a model training method provided in one embodiment of this specification; Figure 3 A flowchart illustrating a content reasoning method provided in one embodiment of this specification; Figure 4 A flowchart illustrating a multimodal content generation method provided in one embodiment of this specification; Figure 5 A flowchart illustrating a reasoning model training method provided in one embodiment of this specification; Figure 6 A flowchart illustrating an image generation method provided in one embodiment of this specification; Figure 7 A flowchart illustrating the processing steps of an image generation method provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification; Figure 9 A schematic diagram of the structure of a content reasoning device provided in one embodiment of this specification; Figure 10 This is a schematic diagram of the structure of a multimodal content generation device provided in one embodiment of this specification; Figure 11 A schematic diagram of the structure of a reasoning model training device provided in one embodiment of this specification; Figure 12 This is a schematic diagram of the structure of an image generation apparatus provided in one embodiment of this specification; Figure 13 This is a structural block diagram of a computing device provided for one embodiment of this specification. Detailed Implementation

[0022] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0023] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0024] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0025] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0026] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0027] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0028] Visual-Language Model (VLM): A multimodal pre-trained model capable of processing both images and text simultaneously.

[0029] Direct Preference Optimization (DPO): A method for directly optimizing the training of language / generative models based on human preference data, without the need for explicit training of reward models.

[0030] Rejection-Sampling Finetuning (RFT): A training strategy that fine-tunes by sampling multiple trajectories and retaining only the positive dominant trajectory.

[0031] Contrastive Language-Image Pre-Training (CLIP): A pre-trained model for image-text comparison, used for matching the relevance of images and text.

[0032] Retrieval-Augmented Generation (RAG): Introduces external retrieval information during the generation process to enhance the model's capabilities.

[0033] Vector Quantization Generative Adversarial Network (VQGAN): A class of image generation models based on vector quantization and generative adversarial networks.

[0034] Diffusion Model: A type of depth generation model that generates images by progressively denoising.

[0035] Knowledge Boundary: refers to the boundary between the parameterized knowledge of the generator and the knowledge required from the external context.

[0036] Concept corruption refers to the phenomenon where unnecessary search results overwrite the generator's existing accurate internal knowledge, leading to a decline in the quality of the generated data.

[0037] Copy Effect: This refers to the generator directly copying the layout and details of the reference image returned by the search, rather than creatively generating based on the reference.

[0038] Long-tail knowledge: Long-tail knowledge refers to knowledge that appears very infrequently in large-scale data but may be important in practical applications.

[0039] Modern text-to-image generation models can produce high-quality images with accurate lighting, coherent structure, and complex scenes, reaching near-saturation on standard benchmarks. However, in real-world user request scenarios, many requests involve knowledge outside the model's training distribution: newly emerging characters and events, regional cultural symbols, niche font typography, historical artifacts, and specific images. Faced with these requests, generators often produce "confident forgeries": beautifully rendered images but completely erroneous content. Through analysis of over 10,000 real user prompts on a production-grade generative AIGC platform, we found that the problem is not a lack of visual synthesis capabilities, but rather a bottleneck in visual knowledge. Generators are trained on fixed corpora and have inherent knowledge cutoffs, while user requests are unbounded, continuously evolving, and deeply tailed. This bottleneck is almost invisible in existing benchmarks because they primarily test prompts within the parameterized knowledge range.

[0040] Search is a natural way to overcome this bottleneck, just as human illustrators consult references before depicting unfamiliar concepts. However, experiments show that a naive search strategy (performing a search for each prompt) not only fails to solve the problem but also introduces new ones: concept erosion (search results overwrite the generator's existing accurate knowledge) and the copy effect (the generator directly copies reference images instead of creatively generating them).

[0041] The root cause is that each search result injects noise into the generator's input space, and current open-source generators cannot distinguish between signals and noise, treating all conditional signals as authoritative input.

[0042] Therefore, we need a generator-aware search design: a search is only performed when the missing knowledge lies outside the generator's knowledge boundary, and the search terms are controlled to enter the generation process. More importantly, the search strategy depends on what the generator knows, and this boundary changes as the generator improves; therefore, the searcher must be jointly calibrated with the generator.

[0043] This specification provides a model training method, and also relates to a content reasoning method, a multimodal content generation method, a reasoning model training method, an image generation method, a model training device, a content reasoning device, a multimodal content generation device, an image generation device, a computing device, a computer-readable storage medium, and a computer program product, to solve the problems mentioned above. Each of these will be described in detail in the following embodiments.

[0044] See Figure 1 , Figure 1 A flowchart of a model training method provided in one embodiment of this specification includes the following steps: Step 102: Obtain sample prompts, target multimodal generation model, and pre-trained inference model.

[0045] Sample prompts are textual information used during the training phase to describe the content of the images to be generated. Specifically, a sample prompt can be a short phrase describing a particular scene, object, or style, or a complete sentence containing multiple entities and their attribute relationships. Sample prompts can be used as input to the inference model to trigger subsequent retrieval processes. Furthermore, sample prompts can carry visual elements, layout constraints, or style preferences related to user needs, and their content can cover various knowledge types such as general objects, entities, cultural symbols, and historical elements.

[0046] A target multimodal generative model is a deep generative model trained to generate corresponding images based on textual prompts. Specifically, a target multimodal generative model can be a diffusion model, a generative adversarial network (GAN), or a vector quantization-based generative model. It can receive conditional textual input and output image data semantically aligned with that text. Furthermore, a target multimodal generative model has a fixed parameterized knowledge boundary, determined by the time cutoff and distribution range of its training data. For knowledge outside the training distribution, the model may not be able to generate images accurately.

[0047] A pre-trained inference model is a visual language model that has been pre-trained through supervised learning and possesses the ability to perform multi-step inference analysis on text prompts. Specifically, a pre-trained inference model may include a Vision-Language Model (VLM) that can simultaneously understand textual semantics and visual concepts. The pre-trained inference model can be used to perform inference tasks such as knowledge gap identification, retrieval query generation, and reference information filtering and integration on input sample prompts. Furthermore, the pre-trained inference model can output a structured inference trajectory, which includes the decision-making process, intermediate results, and the final adjusted prompts.

[0048] In practical applications, sample prompt words are obtained.

[0049] Alternatively, labeled sample prompts can be read from a pre-built training dataset, which may contain de-identified sets of real user requests.

[0050] Another alternative approach is to perform data augmentation and rewriting on the original text in real time, dynamically generating sample prompts. For example, an initial prompt can be replaced with synonyms, transformed in sentence structure, or replaced with entities to generate multiple variants as sample prompts, thereby expanding the diversity of the training data.

[0051] Another option is to first obtain a pre-trained inference model and a target multimodal generation model, then use both to infer and generate a large amount of raw text, and then back-select representative sample prompt words based on the quality assessment results of the generated images, that is, to obtain the sample prompt words required for training through exploratory sampling.

[0052] This step, by acquiring sample prompts, the target multimodal generative model, and the pre-trained inference model, provides complete and reusable foundational data and model components for subsequent collaborative optimization based on inference trajectories and generated images. Specifically, the sample prompts serve as the input source for the task to be processed, the target multimodal generative model acts as the actual executor of image generation, and the pre-trained inference model acts as the controller for decision-making and information integration; together, they constitute the initial state of a closed-loop feedback system. This acquisition process ensures that the inference model can generate different inference trajectories multiple times based on the same prompts in subsequent steps, and the generative model maps each trajectory to a visualized candidate image, thereby achieving effective evaluation and selection of inference quality.

[0053] Step 104: Using a pre-trained inference model, perform multiple inferences based on sample prompts to obtain multiple sample inference trajectories.

[0054] A sample inference trajectory is a structured data sequence generated by a pre-trained inference model during a single complete inference process for a sample prompt word. Specifically, the sample inference trajectory can include candidate knowledge gaps identified after semantic analysis of the sample prompt word during the pre-trained inference model's search decision gating task; the target knowledge gap determined after evaluating the influence of each candidate knowledge gap; a retrieval index for the target modality generated for the target knowledge gap; image information of multiple candidate reference images obtained from an external retrieval system based on the retrieval index; image information of the target reference image selected from the candidate reference images using the principle of information complementarity; and the target prompt word obtained after adjusting the sample prompt word based on the target knowledge gap and the image information of the target reference image. The sample inference trajectory can be used as input conditions for a target multimodal generation model to guide the generation process. Furthermore, different sample inference trajectories correspond to different search decision combinations, such as whether to trigger a search, which references to select, and how to describe the reference information.

[0055] Sample inference trajectories can be used to characterize how a pre-trained inference model, when faced with a specific sample prompt, makes decisions about whether to trigger a search and how to integrate the search results based on its judgment of the knowledge boundary of the target multimodal generative model. Here, the knowledge boundary refers to the set of visual concepts that the target multimodal generative model can accurately generate, while the knowledge gap refers to visual concepts in the sample prompt that exceed this set. Furthermore, different sample inference trajectories can reflect the diverse search decision paths generated by the pre-trained inference model under the same sample prompt due to different strategies; for example, triggering a search in one inference and abandoning the search in another.

[0056] The knowledge boundary is the range of visual concepts covered by the parameterized knowledge of the target multimodal generative model or target image generator. It defines the set of visual knowledge that the generator can accurately generate. Specifically, the knowledge boundary can include all entities, attributes, scenes, styles, and visual relationships learned by the generator from training data during training, which are encoded in the model's weight parameters. The knowledge boundary can be used as a criterion for the inference model to make search decisions. When the visual concepts in the sample prompts or initial content are within the generator's knowledge boundary, the inference model can abandon the search to save computational resources and avoid concept erosion; when the concepts are outside the knowledge boundary, the inference model should trigger a search to obtain external reference information to supplement the missing visual knowledge. The knowledge boundary is not a fixed static set; it expands dynamically as the generator internalizes new visual knowledge into the model parameters through preference optimization training. Therefore, the search strategy of the inference model must be synchronized with the generator's current knowledge boundary to achieve accurate search decisions.

[0057] A knowledge gap is a visual concept in the sample prompts or initial content that exceeds the knowledge boundary of the target multimodal generation model or the target image generator. It represents a missing piece of content that requires external retrieval to obtain supplementary information for accurate generation. Specifically, a knowledge gap can include the appearance details of a specific entity mentioned in the prompts, the spatial relationships between entities, specific material or texture features, or the precise style of cultural symbols. Knowledge gaps can be used as identification and evaluation objects when the inference model performs search decision gating tasks. There is an inherent relationship between knowledge gaps and knowledge boundaries: the identification of knowledge gaps depends on the inference model's judgment of the generator's knowledge boundary. When the inference model determines that a visual concept is outside the generator's knowledge boundary, that concept is marked as a knowledge gap. As the generator's knowledge boundary expands through training, concepts that were originally knowledge gaps may be reclassified within the knowledge boundary, and the inference model will accordingly abandon the search for that concept in subsequent inference.

[0058] Furthermore, knowledge gaps can include, but are not limited to, the following: The first type is a gap where the text encoder has not established a semantic vector at all, that is, the concept has never appeared or rarely appears in the training corpus of the pre-trained text encoder, which makes it impossible for the word to be mapped into an effective semantic vector. The second type is the gap where the text encoder has established semantic vectors but the generator has not established spatial mapping relationships. That is, the model can understand the general meaning of the word, but its cross-attention mechanism has not established a reliable mapping relationship between the semantic position and the specific pixel layout. In this case, the model tends to produce "confident forgeries": the image is beautifully rendered but the key content is completely wrong. The third type is a gap where semantic vectors and spatial mapping relationships have been established but the accuracy is insufficient or the information is outdated. This means that the model has encountered the concept during training, but the frequency is low or the representation is insufficient, resulting in coarse semantic details in the encoding and inaccurate mapping relationships. Alternatively, the true visual style of the concept may have changed after the training data was collected, but the model still stores an outdated version.

[0059] Knowledge gaps and knowledge boundaries are inherently related: the identification of knowledge gaps depends on the inference model's judgment of the generator's knowledge boundaries. When the inference model determines that a visual concept is outside the generator's knowledge boundaries, that concept is marked as a knowledge gap. As the generator's knowledge boundaries expand through training, concepts that were originally knowledge gaps may be reclassified within the knowledge boundaries, and the inference model will correspondingly abandon the search for that concept in subsequent inferences. The inference model can indirectly infer the boundary location by observing the rewards obtained by the generator in generating images of that concept during the co-training phase, and adjust its search decisions accordingly.

[0060] In practical applications, there are several ways to obtain multiple sample reasoning trajectories by using a pre-trained reasoning model to perform multiple inferences based on sample prompts.

[0061] An alternative approach is to repeatedly perform forward propagation of the pre-trained inference model on the same sample prompt word, using the same model parameters and sampling strategy each time, thereby obtaining multiple sample inference trajectories that are similar in probability distribution but may have different specific decision paths.

[0062] Another alternative approach is to introduce different random seeds or adjust the sampling hyperparameters in multiple inferences to explore a wider search decision space, so that the obtained sample inference trajectories include both common decision paths and exploratory marginal decision paths.

[0063] Another alternative approach is to apply a slight perturbation to the sample prompts during each inference, such as randomly inserting or deleting non-key modifiers in the prompts, allowing the pre-trained inference model to perform inference based on the perturbed prompts, thereby obtaining different inference trajectories that are robust to input perturbations.

[0064] This step utilizes a pre-trained inference model to perform multiple independent inferences on the same sample prompt word, obtaining multiple sample inference trajectories. This provides rich and comparable candidate input conditions for subsequent generation and evaluation stages. Thus, the system can automatically generate a set of inference trajectory samples with inherent relative advantages and disadvantages from the same input. This allows for the construction of a data pool for subsequent advantage evaluation and screening without manual annotation, providing an effective foundation for selecting positive advantage samples based on relative advantages within the group.

[0065] Step 106: Using the target multimodal generation model, generate candidate images of samples corresponding to multiple sample inference trajectories based on multiple sample inference trajectories.

[0066] Candidate images are images generated by the target multimodal generative model based on a specific inference trajectory. Specifically, a candidate image can be a pixel matrix with a preset resolution, and its content should correspond to the adjusted cue words in the inference trajectory. Candidate images can be used as the basis for subsequent advantage evaluation to quantify the contribution of the inference trajectory to the generation quality. Furthermore, for multiple inference trajectories of the same cue word, the target multimodal generative model can generate multiple corresponding candidate images.

[0067] In practical applications, there are various ways to generate corresponding candidate images of samples based on multiple sample inference trajectories using a target multimodal generative model.

[0068] An alternative approach is to input the adjusted target cue words contained in each sample inference trajectory into the target multimodal generation model, and then have the target multimodal generation model perform image generation sampling based on the text conditions to output a sample candidate image.

[0069] Another alternative approach is to generate multiple candidate images through multiple random samplings for each sample inference trajectory, and then select one of them as the representative sample candidate image corresponding to that trajectory, in order to reduce the evaluation bias caused by the randomness of a single generation.

[0070] This step utilizes a target multimodal generative model to generate corresponding candidate images for each of the multiple inference trajectories corresponding to the same sample prompt word. This directly maps the decision differences of the inference model into visualized image quality differences, providing an objective and comparable quantitative object for subsequent advantage evaluation. Specifically, whether each sample inference trajectory includes retrieval, the method of selecting retrieval results, and the different prompt word integration strategies are transformed by the generative model into actual performance differences of candidate images in dimensions such as list compliance, text follow-up, and visual reference fidelity. Thus, this step enables the system to automatically obtain a set of images corresponding one-to-one with the inference trajectory from the same original input, establishing a causal relationship between inference decisions and generation quality without manual intervention. This provides a reliable source of evaluation data for selecting positive advantage trajectories based on intra-group relative advantage and ensures that the positive samples used for subsequent supervised fine-tuning training have clear and verifiable quality criteria.

[0071] Step 108: Evaluate the advantage of each candidate image to obtain the relative advantage value of each candidate image.

[0072] Advantage assessment is a process of quantitatively scoring the quality of candidate images based on predefined assessment dimensions and models. Specifically, advantage assessment can employ a pre-trained advantage assessment model, which can be a visual language model capable of scoring images across multiple dimensions based on sample prompts and reference information within the sample inference trajectories. Advantage assessment can be used to generate a relative advantage value for each candidate image, thereby distinguishing the merits of different inference trajectories.

[0073] The relative advantage value is a numerical measure of the quality difference between a candidate image and other candidate images in the same group. Specifically, the relative advantage value can be obtained by calculating the difference between the score of a candidate image and the group's average score, or by sorting and normalizing, among multiple candidate images generated from the same prompt word in the same group. The relative advantage value can be used to indicate whether the corresponding sample inference trajectory positively or negatively affects the generation quality. Furthermore, a positive relative advantage value indicates that the corresponding sample inference trajectory is better than the group average, while a negative value indicates that it is worse than the average.

[0074] In practical applications, there are various specific methods for evaluating the relative advantage value of each candidate image sample.

[0075] An alternative approach is to use a pre-trained advantage evaluation model, such as a visual language model, to score each candidate image individually and obtain an evaluation score. Then, in the score set of all candidate images corresponding to the same prompt word, the difference between each image and the average or median score of that set is calculated, and this difference is used as the relative advantage value.

[0076] Another alternative approach is to construct a pairwise comparison, which combines multiple candidate images corresponding to the same sample prompt word in pairs, uses an advantage evaluation model to determine which image in each pair is better, and then calculates the relative advantage value of each image based on the results of all pairwise comparisons. This relative advantage value reflects the image's relative ranking position within the group.

[0077] Another option is to score each image based on multiple preset evaluation dimensions, such as checklist compliance, rubric score, cue word follow-through, text rendering effect, image artifacts, visual reference fidelity, and text reference fidelity. Then, the scores of each dimension are weighted and summed or directly summed to obtain a comprehensive score. Finally, a relative advantage value is calculated based on the comprehensive score within the same group.

[0078] This step employs a pre-trained dominance evaluation model to assess the quality of multiple candidate images corresponding to the same prompt word, thereby quantifying the generation quality differences caused by different inference trajectories into a sortable and filterable numerical signal. This step enables the system to extract effective contrastive learning signals from multiple generated results from the same input, providing an accurate and reliable selection basis for subsequent supervised fine-tuning based on positive dominance sample inference trajectories, and improving the efficiency and stability of iterative training of the inference model.

[0079] Step 110: Based on the relative advantage value, select the positive advantage sample inference trajectory from multiple sample inference trajectories.

[0080] Positive dominance inference trajectories are those inference trajectories whose corresponding candidate images have a positive relative dominance value. Specifically, positive dominance inference trajectories can be the subset of trajectories selected from multiple inference trajectories that can guide the generation of higher-quality images. Positive dominance inference trajectories can be used as positive samples for supervised fine-tuning training to update the parameters of the pre-trained inference model. Furthermore, positive dominance inference trajectories include correct search decisions, such as accurately identifying the knowledge gaps to be retrieved, selecting complementary references, reasonably integrating reference information, or reasonably abandoning knowledge gaps that do not need to be retrieved.

[0081] In practical applications, there are multiple ways to select positive advantage sample inference trajectories based on relative advantage values.

[0082] An alternative approach is to directly select all sample inference trajectories with a relative advantage value greater than zero as positive advantage sample inference trajectories, that is, all trajectories that are better than the average level of the same group are retained.

[0083] Another alternative approach is to set a positive threshold and retain only the inference trajectories of samples whose relative advantage value exceeds the threshold, thereby selecting only trajectories that are significantly better than the average level of the group and filtering out marginal trajectories that are only slightly better than the average level.

[0084] Another option is to sort the samples within each group by relative advantage value from high to low, and retain only the inference trajectories of the top K samples as positive advantage inference trajectories. In this way, each group only takes one or a few of the top-ranked trajectories, making the training signal more focused on the correct decision-making pattern.

[0085] This step, based on the relative advantage value of each candidate image, selects those trajectories with positive relative advantage values ​​from multiple inference trajectories corresponding to the same prompt word as positive advantage sample inference trajectories, thereby automatically identifying inference decision sequences superior to those in the same group. Positive advantage sample inference trajectories can accurately trigger necessary retrieval, effectively filter noisy references, and reasonably abandon meaningless retrieval, guiding the target multimodal generation model to produce higher-quality candidate images. Therefore, this step directly transforms the results of intra-group relative comparisons into a set of positive samples that can be used for model updates, providing a precise, consistent, and scalable source of training data for subsequent supervised fine-tuning based on positive advantage sample inference trajectories.

[0086] Step 112: Based on the positive dominance sample inference trajectory, perform supervised fine-tuning training on the pre-trained inference model to obtain the target inference model.

[0087] Supervised fine-tuning is the process of updating the parameters of a pre-trained model using labeled positive examples. Specifically, supervised fine-tuning can employ the cross-entropy loss function, using the decision sequences in the reasoning trajectories of positively dominant samples as supervisory signals to optimize the output distribution of the pre-trained inference model. Supervised fine-tuning can enable the inference model to learn which search decisions produce high-quality generated results, thereby improving its inference ability in assisting the generation process of multimodal generative models. One possible example is the reinforcement fine-tuning (RFT) paradigm.

[0088] The target inference model is a reasoning model that, after supervised fine-tuning training, can adaptively perform inference based on input prompts and the target multimodal generative model. Specifically, in practical applications, the target inference model can be used to perform gating, reference filtering, and integration on the prompts to be generated, outputting adjusted reference prompts. The knowledge boundary of the target inference model matches the knowledge boundary of the target multimodal generative model used during training, enabling it to naturally learn from the reward signal when to trigger a search and when to abandon the search without explicit boundary labels.

[0089] This specification describes an embodiment that acquires sample prompts, a target multimodal generation model, and a pre-trained inference model. The pre-trained inference model performs multiple inferences based on the sample prompts to obtain multiple sample inference trajectories, automatically generating diverse sample inference trajectories without manual annotation. The target multimodal generation model generates corresponding sample candidate images based on each sample inference trajectory, quantifying the actual generation quality of each trajectory through image representation. By evaluating the relative advantage of each candidate image, the relative advantage value is obtained, allowing inference trajectories from different prompts to be compared on a uniform relative scale, accurately distinguishing which inference trajectories produce superior generation results. Based on this relative advantage value, positively dominant sample inference trajectories are selected from the multiple inference trajectories, retaining those that can guide the generation of higher-quality images. The pre-trained inference model is then supervised and fine-tuned based on these positively dominant sample inference trajectories, enabling it to naturally learn the generation capabilities of the target multimodal generation model during training without explicit annotation, and providing reasonable inferences for the sample prompts. This achieves calibration between the target inference model's search strategy and the target multimodal generation model's capabilities. In other words, the inference model can naturally learn the knowledge boundary of the target multimodal generative model from the rewards, without any explicit knowledge boundary labels, thus achieving joint calibration of the target inference model's search strategy and generator capabilities.

[0090] In one optional embodiment of this specification, the pre-training steps of the pre-trained inference model include: Obtain pre-trained prompt words, the label inference trajectories corresponding to the pre-trained prompt words, and the initial inference model; Using the initial inference model, perform at least one task inference based on pre-trained prompts to obtain the initial inference trajectory; Based on the initial inference trajectory and the labeled inference trajectory, the initial inference model is pre-trained under supervision to obtain a pre-trained inference model.

[0091] Pre-training prompts are textual information describing the expected image generation content used in the pre-training phase of the initial inference model. Their structure is similar to or identical to that of sample prompts. Specifically, pre-training prompts can come from a pre-built pre-training dataset, where each pre-training prompt is accompanied by manually labeled or automatically generated labeled inference trajectories. Pre-training prompts can be used as input to the initial inference model, triggering the model to perform inference tasks such as search decision gating, reference filtering, and reference integration. Furthermore, pre-training prompts can cover various knowledge types and difficulty levels to enable the pre-trained model to have a broad knowledge gap identification capability.

[0092] A label inference trajectory is a pre-annotated sequence of structured inference results expected to be output by the inference model for a given pre-training prompt. Specifically, the label inference trajectory may include the candidate knowledge gaps for labels to be identified for the pre-training prompt, the target knowledge gaps for the labels, the image information of the target image to be retrieved, and the target prompts for the labels to be generated. The label inference trajectory can be used as a supervisory signal, compared with the initial inference trajectory actually output by the initial inference model during pre-training. Furthermore, the label inference trajectory can be annotated by human experts based on the pre-training prompt and pre-defined inference rules, or it can be obtained through an automated annotation process, such as generating it using a more powerful teacher model and then manually verifying it.

[0093] The initial inference model is a visual language model that has not been pre-trained or has only undergone basic initialization and needs to learn inference capabilities during the pre-training phase. The initial inference model can be a general pre-trained visual language model with basic text understanding and generation capabilities, but it has not been specifically optimized for tasks such as search decision gating, reference filtering, and reference integration, nor can it perceive the knowledge gaps or boundaries of a specific generator or generation model. The initial inference model can be used to receive pre-training prompts and output initial inference trajectories during pre-training. Furthermore, the parameters of the initial inference model are updated according to the loss function during pre-training, gradually optimizing towards outputting results that closely resemble the label inference trajectories.

[0094] The initial inference trajectory is the sequence of inference results actually output by the initial inference model for a pre-trained prompt word. Specifically, the initial inference trajectory may include candidate knowledge gaps identified by the initial inference model, the target knowledge gap determined from the candidate knowledge gaps, image information of retrieved candidate reference images, image information of the target reference image selected from multiple candidate reference images, and the target prompt word obtained after adjusting the pre-trained prompt word based on the target knowledge gap and the target reference image. The initial inference trajectory can be used to calculate the loss with the labeled inference trajectory, thereby generating the loss signal required for supervised pre-training. Furthermore, the initial inference trajectory may differ significantly from the labeled inference trajectory in the early stages of pre-training, gradually approaching the labeled inference trajectory as training progresses.

[0095] Supervised pre-training is a training process that uses labeled inference trajectories as supervisory signals to update the parameters of an initial inference model. Specifically, supervised pre-training can employ a sequence-to-sequence learning framework, using pre-training prompts as encoder input and labeled inference trajectories as the target output of the decoder. It calculates the cross-entropy loss or word-level matching loss between the initial inference trajectory actually output by the initial inference model and the labeled inference trajectory, and updates the parameters of the initial inference model using backpropagation and gradient descent algorithms. Supervised pre-training can be used to enable the initial inference model to learn standardized inference patterns on a large amount of labeled data, thus obtaining a pre-trained inference model. Furthermore, after the supervised pre-training stage, the pre-trained inference model possesses structured knowledge gap analysis and search decision-making capabilities, but it has not yet been co-calibrated with a specific target multimodal generative model. One possible example is that supervised pre-training can be performed using supervised fine-tuning (SFT).

[0096] In practical applications, there are multiple ways to obtain an initial inference trajectory by using an initial inference model to perform at least one task inference based on pre-trained prompts.

[0097] An alternative approach is to use pre-trained prompts as input, allowing the initial inference model to sequentially perform a search decision gating task, a reference filtering task, and a reference integration task. Each task outputs corresponding intermediate results, and finally, all intermediate results are concatenated or packaged into a structured initial inference trajectory. Specifically, the model first outputs the target knowledge gap, then outputs the image information of the retrieved candidate reference images and the image information of the selected target reference image based on the knowledge gap, and finally outputs the adjusted target prompts. The outputs of these three stages together constitute the initial inference trajectory.

[0098] Another alternative approach is to perform only a portion of the three tasks mentioned above. For example, in some simplified pre-training scenarios, only the search decision gating task can be performed initially, stopping after abandoning candidate knowledge gaps, and using the abandoned candidate knowledge gaps as the initial inference trajectory.

[0099] In practical applications, there are multiple ways to obtain a pre-trained inference model by supervising the initial inference model based on the initial inference trajectory and the label inference trajectory.

[0100] Alternatively, both the initial inference trajectory and the label inference trajectory can be treated as serialized label sequences, and the cross-entropy loss between the two sequences can be calculated. Specifically, for each predicted label in the initial inference trajectory, the negative log-likelihood of its position relative to the corresponding label in the label inference trajectory is calculated, and the average loss across all positions is used as the overall loss to update the parameters of the initial inference model through backpropagation.

[0101] Another alternative approach is to calculate the losses for each of the three tasks separately and then sum them using weighted averages. For example, for the search decision gating task, calculate the classification loss between the predicted knowledge gap and the label knowledge gap; for the reference filtering task, calculate the matching loss between the predicted reference image identifier and the label reference image identifier; for the reference integration task, calculate the sequence generation loss between the generated target prompt and the label target prompt; and then sum these three losses according to preset weights to obtain the total loss for optimization.

[0102] Another option is to introduce the idea of ​​contrastive learning, which maximizes the similarity between positive sample pairs (initial inference trajectory and corresponding label inference trajectory) and minimizes the similarity between negative sample pairs (initial inference trajectory and other mismatched label inference trajectories), and pre-trains using a contrastive loss function.

[0103] This specification's embodiments utilize an initial inference model to perform at least one task inference based on pre-trained prompts and obtain an initial inference trajectory. This allows the model to perform exploratory outputs, providing comparable prediction results for subsequent loss calculations. Supervised pre-training is then performed on the initial inference trajectory and the labeled inference trajectory. This allows the model to gradually learn inference patterns conforming to annotation specifications by minimizing the differences between the two, including accurately identifying which knowledge gaps are worth retrieving, how to select complementary references in the retrieval results, and how to transform reference information into natural semantic descriptions. The pre-trained inference model obtained through the above supervised pre-training possesses structured task inference capabilities and can automatically output a complete inference trajectory including multiple task inference processes.

[0104] In one optional embodiment of this specification, at least one task includes a search decision gating task, a reference filtering task, and a reference integration task. Using the initial inference model, at least one task inference is performed based on pre-trained prompts to obtain the initial inference trajectory, including: A pre-trained inference model is used to perform a search decision gating task. Multiple candidate knowledge gaps are obtained based on pre-trained prompts, and the target knowledge gap is determined from the candidate knowledge gaps. A pre-trained inference model is used to perform a reference filtering task. Based on the target knowledge gap, image information of multiple candidate reference images is retrieved, and image information of the target reference image is determined from the image information of multiple candidate reference images. A pre-trained inference model is used to perform a reference integration task. The pre-trained prompt words are adjusted based on the target knowledge gap and the image information of the target reference image to obtain the target prompt words. The initial inference trajectory is determined based on the target knowledge gap, the image information of the target reference image, and the target prompt words.

[0105] The search decision gating task is a decision-making process in which the inference model analyzes input prompts and determines whether there are worthwhile knowledge gaps to retrieve. Specifically, the search decision gating task may include semantic analysis of prompts to identify multiple candidate knowledge gaps, evaluating the impact of each candidate knowledge gap on image generation, and selecting target knowledge gaps from the candidate knowledge gaps whose impact reaches a preset threshold. The search decision gating task can be used to decide whether to trigger subsequent retrieval operations and which knowledge gaps to retrieve, thereby avoiding unnecessary searches for concepts that the generator already possesses accurate knowledge of, and preventing concept erosion. Furthermore, the output of the search decision gating task may include zero or one or more target knowledge gaps. When there are no target knowledge gaps, the inference model can directly skip subsequent reference filtering and reference integration tasks and directly generate an inference trajectory based on the abandoned candidate knowledge gaps and the original prompts.

[0106] The reference filtering task is the process of selecting the most suitable reference image information from multiple candidate reference images returned by retrieval, based on the type and content of the knowledge gap after identifying the target knowledge gap. Specifically, the reference filtering task may include generating a retrieval index based on the target knowledge gap, calling a retrieval interface to obtain image information from multiple candidate reference images, and determining the image information of the target reference image from the image information of the multiple candidate reference images based on the principle of information complementarity. The reference filtering task can be used to select references that can fill a specific knowledge gap while minimizing irrelevant content, avoiding the selection of reference images with excessively high overall matching to prevent the copy effect. The reference filtering task can process retrieval results in both image and text modalities simultaneously, and select single-modal or multi-modal reference information based on the type of knowledge gap.

[0107] Reference integration is the process of synthesizing original cue words, target knowledge gaps, and image information from the target reference image into a single enhanced text specification. Specifically, reference integration may involve extracting key visual attributes from the target reference image using natural language descriptions and embedding these descriptions into the original cue words to generate a target cue word containing usage guidance information about the reference image. Reference integration can be used to transform pixel-level visual references into semantic-level text descriptions, thereby forcing semantic extraction and discarding pixel-level noise by using natural language as an information bottleneck, avoiding spatial distortion, identity mixing, and style contamination. Furthermore, the target cue word may include an explicit indicator field to distinguish between attributes that need to be retained from the reference and those that can be freely generated.

[0108] Candidate knowledge gaps are visual concepts or entity information identified through semantic analysis from pre-trained prompts that may exceed the parameterized knowledge scope of the generator. Specifically, candidate knowledge gaps can include details of a particular person's appearance, specific styles of cultural symbols, precise forms of historical artifacts, and typographical features of niche fonts. Candidate knowledge gaps can be used as screening targets in search decision gating tasks, with each gap corresponding to a knowledge type, influence score, and reason for the gap. Furthermore, the number of candidate knowledge gaps can vary depending on the complexity of the prompts; for example, a prompt containing multiple entities may generate multiple candidate knowledge gaps.

[0109] Target knowledge gaps are those selected from candidate knowledge gaps that are deemed worthy of triggering a search operation. These are typically key concepts that significantly impact the quality of image generation. Specifically, target knowledge gaps can have associated knowledge types, such as visual identity, spatial relationships, or text attributes, and corresponding retrieval modalities, such as image search or text search. Target knowledge gaps can serve as the basis for subsequent reference filtering tasks and directly determine the quality of the retrieval query and the relevance of the retrieval results. Furthermore, the number of target knowledge gaps can be zero or one or more, but to control retrieval costs, in practice, it is usually limited to a maximum of three.

[0110] The image information of candidate reference images consists of multiple reference images and their associated metadata returned by the retrieval interface that are related to the target knowledge gap. Specifically, the image information of candidate reference images may include the image's pixel data, image thumbnail, image source URL, image title description, and the similarity score between the image and the retrieval index. The image information of candidate reference images may carry information noise, which refers to irrelevant or redundant content with a matching degree to the retrieval index that is below a preset threshold. Furthermore, the image information of candidate reference images may contain complete references with high overall matching degree and complementary references with high local matching degree.

[0111] The image information of the target reference image is the top-ranked reference content selected from multiple candidate reference images through a reference filtering task to fill the target knowledge gap. Specifically, the image information of the target reference image may only include visual attributes directly related to the knowledge gap, such as the color scheme of a character or the geometry of a flag, while discarding information noise, background, or compositional information unrelated to the knowledge gap, such as image watermarks. The image information of the target reference image can be used as input for a reference integration task, where the visual information is converted into natural language descriptions to be incorporated into the target cue words. Furthermore, the image information of the target reference image can be image references, text references, or both, depending on the modal requirements of the knowledge gap.

[0112] Target prompts are enhanced text specifications output from the reference integration task, supplemented by knowledge gap filling and reference information fusion, and are used as input to a multimodal generative model to generate images. Specifically, target prompts may include complete retention or rewriting of the original prompts, explicit identification of target knowledge gaps, semantic description of image information of the target reference image, and guidance on the use of image information, fusion strategies, and free generation instruction fields. Target prompts can be used to replace the original prompts as conditional input to the generator, enabling the generator to acquire previously missing external knowledge without being directly exposed to pixel-level noise. For example, the reference information described in the target prompts can be presented in a semantic form such as "the character is wearing a cyan-gold asymmetrical lapel robe," rather than directly pasting the binary data of the reference image.

[0113] In practical applications, there are multiple ways to perform search decision gating tasks and identify target knowledge gaps.

[0114] An alternative approach involves using a pre-trained inference model to perform full-text semantic analysis on pre-trained prompts, identifying the specific entities, attribute descriptions, and relational constraints involved. Concepts with low confidence or complete absence in the model's parameterized knowledge are then marked as candidate knowledge gaps. Specifically, the model can output a list of candidate knowledge gaps, each accompanied by an influence score, and then select gaps with influence scores exceeding a preset threshold as target knowledge gaps.

[0115] Another option is to incorporate historical performance data of the generator into the gating decision. For example, a pre-trained inference model can call a lightweight generator capability evaluation module to quickly assess whether the current generator can accurately generate the concept for each candidate knowledge gap, and only include the gap in the target knowledge gap if the generator is judged to be unable to generate it accurately.

[0116] In practical applications, there are multiple ways to perform reference filtering tasks and determine the image information of the target reference image.

[0117] One alternative approach is to determine the retrieval index for the target modality based on the type of knowledge gap, such as an image search index or a webpage text search index. This involves calling a retrieval interface to obtain image information from multiple candidate reference images or webpage text, then calculating the similarity score between the image information or webpage text of each candidate reference image and the retrieval index. Simultaneously, the information noise level carried by each candidate reference image is analyzed, and candidates with moderate similarity and low information noise are selected as the target reference images. Specifically, reference images with excessively high similarity are easily copied directly by the model and should be excluded; reference images with excessively low similarity are irrelevant to the knowledge gap and should also be excluded; only complementary references with moderate similarity that only provide the missing attribute are selected.

[0118] Another alternative approach is to use the principle of information complementarity for multi-reference fusion. For the same target knowledge gap, image information from multiple candidate reference images can be selected simultaneously. Each reference provides different aspects of the gap, and then these multi-source reference information are merged in the integration phase.

[0119] Another alternative approach is to introduce a visual language model to perform content understanding on candidate reference images, extract visual elements in each reference image that are directly related to the target knowledge gap, and use the extracted local visual descriptions as reference information while discarding background or pose information that is irrelevant to the gap, thereby further reducing noise.

[0120] In practical applications, there are multiple ways to perform reference integration tasks and obtain target prompts.

[0121] An alternative approach is to use the original pre-trained prompts as a base and insert phrases or clauses derived from semantically describing the image information of the target reference image at the locations in the prompts where knowledge gaps need to be filled. For example, if the original prompt is "a warrior stands on the battlefield," the target knowledge gap is "the style of the warrior's armor," and the target reference image provides information about "a blue-gold asymmetrical lapel robe," then the integrated target prompt could be "a warrior wearing a blue-gold asymmetrical lapel robe stands on the battlefield."

[0122] Another alternative approach is to add a separate field containing instructions on the use of image information to the target cue. This field explicitly lists, in natural language, the specific attributes that need to be borrowed from the reference image, while retaining other freely generated parts of the original cue. For example, the target cue could be constructed as: "Basic description: A warrior stands on the battlefield. Reference borrowing: The warrior's armor must use the colors (cyan-gold) and collar design (asymmetrical lapel) from the reference image. The rest is up to you." This approach distinguishes between constraints and freedom through explicit borrowing instructions, further suppressing the copying effect.

[0123] Another alternative approach is to transform the key visual attributes of the target reference image into a set of discrete visual labels, and then append these labels as metadata to the cue words. The generator has already learned how to correctly parse these labels during training.

[0124] This specification's embodiments utilize a pre-trained inference model to sequentially execute search decision gating, reference filtering, and reference integration tasks. This allows the inference model to learn the complete inference process from pre-trained prompts to target prompts. Specifically, the gating task identifies multiple candidate knowledge gaps based on pre-trained prompts and selects the target knowledge gap. This mechanism ensures that subsequent retrieval is triggered only when the knowledge gap is determined to have a substantial impact on image generation, thus avoiding concept erosion caused by indiscriminate retrieval. The filtering task retrieves image information from multiple candidate reference images based on the target knowledge gap and determines the image information of the target reference image. This process selects references that can fill specific gaps and contain the least irrelevant content based on the principle of information complementarity, effectively preventing the copy effect caused by selecting reference images with excessively high overall matching. The integration task adjusts the pre-trained prompts based on the target knowledge gap and the image information of the target reference image to obtain the target prompt. This process routes visual references through natural language descriptions, forces semantic extraction, and discards pixel-level noise, thereby avoiding modality-specific noise such as spatial distortion, identity mixing, and style contamination that may result from direct pixel injection. Based on the initial reasoning trajectory determined by the target knowledge gap, the image information of the target reference image, and the target cue words, the decision-making process and output results of the reasoning model were fully recorded.

[0125] In one optional embodiment of this specification, the knowledge gap corresponds to a knowledge type, and any knowledge type corresponds to a different modality of retrieval index; Before retrieving image information from multiple candidate reference images based on the target knowledge gap, the process also includes: Using a pre-trained inference model, a retrieval index for the target modality corresponding to the target knowledge gap is generated based on the knowledge type of the target knowledge gap. Image information from multiple candidate reference images is retrieved based on the target knowledge gap, including: Based on the retrieval index of the target modality, retrieve image information of multiple candidate reference images of the target modality.

[0126] Knowledge types are attribute information used to characterize the content category to which a knowledge gap belongs. Different knowledge types correspond to different information retrieval modalities. Specifically, knowledge types can include visual identity categories, such as character features, clothing styles, and object appearance; spatial geometry categories, such as relative position, proportions, and structural layout; texture and material categories, such as surface texture, reflective properties, and patterns; and textual symbol categories, such as font style, logo design, and text arrangement. Knowledge types can be used to determine whether an image modality, text modality, or multimodal fusion retrieval index should be used during retrieval. Knowledge types can be identified simultaneously during the identification of knowledge gaps in the pre-trained inference model; that is, when acquiring knowledge gaps, the knowledge type of the knowledge gap is identified and labeled simultaneously to assist in the subsequent generation of retrieval indexes for different modalities corresponding to the knowledge gap.

[0127] A retrieval index is a query key or search condition used to find reference information related to a target knowledge gap from a retrieval database. Different modalities of retrieval indexes have different data formats and query interfaces. Specifically, a retrieval index can be search keywords represented in text form, similarity matching vectors represented as image feature vectors, or composite query conditions that integrate text and image information. The retrieval index can be used as input to a retrieval system to recall candidate reference content from the corresponding retrieval database. Furthermore, for image modal retrieval indexes, it can be a text query describing the target knowledge gap, such as query terms from an image search engine or a feature embedding of an example image; for text modal retrieval indexes, it can be a natural language description or a set of keywords.

[0128] The target modality is the information retrieval channel type corresponding to the retrieval index, and it has a mapping relationship with the type of knowledge gap in the target knowledge gap. Specifically, the target modality can be a pure image retrieval modality, a pure text retrieval modality, or a hybrid image-text retrieval modality. The target modality can be used to determine which retrieval interface to call and how to parse the returned results. Furthermore, when the knowledge type is a visual identity type, the target modality can be set to an image retrieval modality because visual details are more accurately expressed through images; when the knowledge type is a text symbol type, the target modality can be set to a text retrieval modality because text descriptions are more precise and less prone to ambiguity; when the knowledge type requires both visual and textual information, the target modality can be a hybrid image-text retrieval modality.

[0129] In practical applications, there are multiple ways to generate a retrieval index for the target modality based on the knowledge type of the target knowledge gap.

[0130] One alternative approach is to pre-establish a mapping table between knowledge types and retrieval modalities. After the pre-trained inference model identifies the target knowledge gap, it extracts the knowledge type label corresponding to the knowledge gap, then queries the mapping table to obtain the corresponding target modality, and finally constructs a retrieval index for that modality based on the specific content of the target knowledge gap.

[0131] Alternatively, a pre-trained inference model can directly output the target modality and retrieval index through end-to-end learning, without relying on an explicit mapping table. During inference, the model can automatically generate a suitable retrieval index format based on the input target knowledge gaps.

[0132] This specification's embodiments pre-associate knowledge types with knowledge gaps and establish a correspondence between knowledge types and retrieval modalities, enabling different types of knowledge gaps to automatically adapt to the most suitable information retrieval channels. Specifically, a pre-trained inference model is used to generate a retrieval index for the target modality based on the knowledge type of the target knowledge gap, avoiding the incomplete and mismatched retrieval results caused by using a uniform retrieval modality for all knowledge gaps with a single retrieval modality. Furthermore, image information of multiple candidate reference images for the target modality is retrieved based on the retrieval index of the target modality, ensuring that the content returned by the retrieval matches the essential requirements of the knowledge gap. This provides higher-quality input for subsequent reference filtering and integration tasks, thereby improving the efficiency and accuracy of the entire search enhancement generation process.

[0133] In one optional embodiment of this specification, the image information carries information noise, which is information whose matching degree with the retrieval index is lower than a preset threshold. From the image information of multiple candidate reference images, the image information of the target reference image is determined, including: Based on the retrieval index of the target modality, the similarity between the image information of multiple candidate reference images, and the information noise of multiple candidate reference images, the image information of the target reference image is determined from the image information of multiple candidate reference images.

[0134] Information noise refers to content components in image information that have a low matching degree with the retrieval index, are irrelevant to the target knowledge gap, or are redundant. Specifically, information noise can include visual regions in candidate reference images that have a matching degree with the retrieval index below a preset threshold, color or texture information that does not match the target knowledge gap, and image watermarks in reference images that exceed the required range, etc. Information noise can be used to measure the proportion of interfering components carried in the image information of a candidate reference image. Higher information noise indicates that the reference image contains more irrelevant content, and is more likely to have a negative impact on the generator. Furthermore, information noise and similarity are not simply complementary; an image may simultaneously have high similarity and high information noise.

[0135] The similarity between the image information of the retrieval index and the candidate reference image is a quantitative metric measuring the degree of matching between the candidate reference image and the target knowledge gap. Specifically, similarity can be calculated based on the cosine distance between feature vectors, the relevance score between the image and the text query, or the matching probability output by the visual language model. Similarity can be used to characterize the proportion of signals directly related to the target knowledge gap contained in the image information of the candidate reference image; a higher similarity indicates a stronger semantic association between the reference image and the target knowledge gap. Furthermore, similarity can be used in conjunction with information noise; similarity reflects signal strength, while information noise reflects interference strength, and both together determine whether a candidate reference image is suitable as a target reference image.

[0136] A preset threshold is a numerical boundary used to distinguish between valid signals and information noise; image information below this threshold is considered noise. Specifically, the preset threshold can be a fixed empirical threshold or a dynamically calculated adaptive threshold. The preset threshold provides an objective cutoff point in the information noise determination process, enabling the pre-trained inference model to uniformly determine which image information content is redundant or irrelevant.

[0137] In practical applications, there are various ways to implement filtering based on similarity and information noise.

[0138] One alternative approach is to calculate the similarity between the image information of each candidate reference image and the retrieval index of the target modality, while simultaneously evaluating the information noise level of the candidate reference image. Then, the candidate reference images are ranked according to a weighted composite score of similarity and information noise, and the candidate with the highest composite score is selected as the target reference image. For example, the composite score could be set as the similarity minus the information noise coefficient multiplied by the noise level, allowing references with high similarity but also high noise to be excluded, while references with moderate similarity but extremely low noise may be selected.

[0139] Another alternative approach is to employ a two-stage screening strategy: first, select candidate reference images whose similarity falls within a preset range based on similarity, excluding those with excessively low or high similarity; then, select one or more of the remaining candidates with the lowest information noise level as the target reference image.

[0140] In addition, a reference selection model can be trained using reinforcement learning. This model uses similarity and information noise as state features, outputs a selection action, and optimizes the model using the quality of subsequently generated images as a reward signal, so that the reference selection strategy can adaptively balance similarity and noise.

[0141] This specification's embodiments achieve a refined reference filtering mechanism by simultaneously considering the similarity between the retrieval index of the target modality and the image information of the candidate reference images, as well as the information noise of the candidate reference images. Filtering candidate reference images based on similarity ensures information complementarity; the reference images effectively supplement the image generation process. At the same time, reference images may also carry a large amount of noise, which can be eliminated through joint filtering based on similarity and noise, providing a clean input source for addressing the natural language information bottleneck in subsequent integration tasks.

[0142] One optional embodiment of this specification utilizes a pre-trained inference model to perform a search decision gating task, obtaining multiple candidate knowledge gaps based on pre-trained prompts, and determining the target knowledge gap from the candidate knowledge gaps, including: Using a pre-trained inference model, semantic analysis is performed on the pre-trained prompt words to obtain the semantic analysis results; Based on the semantic analysis results, at least one candidate knowledge gap is identified; The target knowledge gap is determined based on the influence of at least one candidate knowledge gap, where the influence is the degree to which the knowledge gap affects image generation.

[0143] Semantic analysis results are structured semantic information extracted from pre-trained prompts using natural language understanding. These results characterize the entities, attributes, relationships, and importance contained within the prompts. Specifically, semantic analysis results can include subject entities such as people, objects, and scenes; modifiers for each entity such as color, shape, and material; spatial or logical relationships between entities such as location, action, and subordination; and the weight or salience of each semantic unit within the entire prompt. Semantic analysis results can be used as a basis for subsequent identification of candidate knowledge gaps, where semantic units that are missing or have low confidence in the model's parameterized knowledge are marked as potential knowledge gaps. Furthermore, semantic analysis results can be obtained from the encoder output of a visual language model, which can be represented as a latent vector sequence or attention weight distribution, or further output as text or structured data.

[0144] Candidate knowledge gaps are sets of semantic units or concepts identified from pre-trained prompts based on semantic analysis results, which may exceed the parameterized knowledge scope of the generator or generative model. Specifically, candidate knowledge gaps can include a specific entity, an attribute of that entity, or a relationship between entities from the semantic analysis results. Candidate knowledge gaps can be used as screening targets for subsequent gating tasks, and each candidate knowledge gap can be associated with an influence score.

[0145] Impact is a quantitative metric used to measure the degree to which a candidate knowledge gap affects the quality of the generated image. It characterizes the magnitude of the change in the generated image quality caused by the correct or incorrect handling of the knowledge gap. Specifically, a higher impact value indicates that the knowledge gap is more critical to the correctness, completeness, and aesthetic quality of the final image; a knowledge gap with zero impact value means that even if completely ignored, it will not affect the acceptability of the generated image. Impact can be used as a ranking or screening criterion for selecting target knowledge gaps from multiple candidate knowledge gaps. Furthermore, the determination of impact can be based on the type of knowledge gap, the grammatical role of the knowledge gap in the prompt words, and the generator's historical performance data on this type of concept, etc.

[0146] In practical applications, there are multiple ways to determine at least one candidate knowledge gap based on semantic analysis results.

[0147] One alternative approach is to extract all entity nouns and proper nouns from the semantic analysis results and use these nouns as candidate knowledge gaps. Specifically, if the pre-training prompts contain nouns with specific referents, such as "a certain character," "a certain historical figure," or "a certain cultural symbol," these nouns are generally outside the parameterized knowledge range of the general generator and should be marked as candidate knowledge gaps.

[0148] Another alternative approach is to identify components in the semantic analysis results that contain uncertain modifiers or vague descriptions, and then use the semantic units modified by these components as candidate knowledge gaps. For example, when prompts contain words such as "unknown," "something," or "similar to," it indicates that the user expects the generator to supplement specific information, and the corresponding concepts should be considered as candidate knowledge gaps.

[0149] Another alternative approach is to determine whether each semantic unit belongs to the knowledge already known to the generator based on generator capability assessment. The pre-trained inference model can quickly predict whether the generator can accurately generate the content for each entity or attribute in the semantic analysis results based on the generator capability assessment, identifying units predicted as "unable to generate" or "low generation confidence" as candidate knowledge gaps.

[0150] This specification's embodiments obtain semantic analysis results by performing semantic analysis on pre-trained prompts. This transforms the fuzzy natural language expressions in the pre-trained prompts into a structured set of semantic units, enabling precise semantic understanding-based localization of knowledge gaps. The target knowledge gap is determined based on the influence of each candidate knowledge gap. This makes gating decisions no longer a binary matter of whether to trigger a search, but rather allows for refined filtering based on the actual impact of each knowledge gap on image generation. Specifically, if the generator already knows the concept, it will be determined as not requiring retrieval during generator capability evaluation, thus preventing a search. Conversely, concepts unknown to the generator but with low influence will also not trigger a search, reducing the probability of noise injection and preventing concept erosion. In other words, the gating task controls the quantity and quality of knowledge gaps entering the subsequent retrieval process at the source, ensuring that each search has a clear basis for value.

[0151] One optional embodiment of this specification utilizes a pre-trained inference model to perform a reference integration task, adjusting pre-trained prompts based on target knowledge gaps and image information from the target reference image to obtain target prompts, including: A pre-trained inference model is used to perform a reference integration task. Based on semantic analysis results, target knowledge gaps, and image information of the target reference image, the pre-trained prompt words are adjusted to obtain target prompt words. The target prompt words include guidance information on the use of image information of the target reference image.

[0152] The reference integration task is the process by which an inference model synthesizes the original cue words, semantic analysis results, target knowledge gaps, and image information from the target reference image into a single enhanced text specification. Specifically, the reference integration task may include extracting visual attributes directly related to the target knowledge gap from the image information of the target reference image, embedding these attributes into appropriate positions in the original cue words using natural language descriptions, and generating a final cue word containing usage guidance information related to the reference image. The reference integration task can be used to transform reference information into semantic conditions that the generator can correctly understand, avoiding spatial distortion, identity mixing, and style pollution caused by directly injecting pixel data. Furthermore, the usage guidance information in the target cue word output by the reference integration task can be presented in the form of explicit fields (such as "Reference:...") or implicitly integrated.

[0153] Semantic analysis results are the structured semantic information extracted from pre-trained prompts using natural language understanding. This includes entities, attributes, relationships, and the grammatical roles and importance weights of each semantic unit within the prompt. Specifically, semantic analysis results can characterize entities within a prompt, their modifying attributes, spatial or logical relationships between entities, and the importance of each semantic unit relative to the entire sentence. Semantic analysis results can be used in reference integration tasks to determine where to insert reference information into prompts and at what granularity. Furthermore, semantic analysis results can indicate which semantic units the generator has already correctly generated, allowing for integration without modifying these parts, only supplementing missing knowledge.

[0154] Usage guidance information is instructional text embedded in target prompts that instructs the generator on how to process reference image information. Specifically, usage guidance information may include a "borrow reference" field, explicitly listing attributes to be retained from the reference image; a "constraint type" identifier, where hard constraints must be strictly followed and soft constraints suggest referencing; and a "freely generated range" description, indicating which parts can be freely generated by the generator. Usage guidance information can be used to enable controlled learning of reference information at the generator level, preventing the generator from indiscriminately copying the overall layout and details of the reference image, thereby suppressing the copying effect. Usage guidance information can be expressed in the form of structured tags or natural language clauses.

[0155] This embodiment of the specification adjusts pre-trained prompts by utilizing semantic analysis results, target knowledge gaps, and image information from the target reference image. This transforms the integration process from a simple splicing of reference information to precisely embedding it into the most needed locations based on an understanding of the prompts' semantic structure. Semantic analysis provides the grammatical skeleton of the original prompts, ensuring a natural syntactic and semantic integration between the inserted reference description and the context. Target knowledge gaps clarify the content to be filled, allowing the integration operation to focus on missing key attributes rather than completely copying the reference image. The image information from the target reference image provides the signal source for internal supplementation while avoiding pixel-level noise propagation. The generated target prompts contain usage guidance information, explicitly or implicitly informing the generator which attributes must be borrowed from the reference image and which parts can be freely developed. This supplements missing visual knowledge while preserving creative space for the generator. The generator no longer treats the reference image as a template to be copied entirely, but rather as a knowledge source providing specific attributes, effectively suppressing the copying effect.

[0156] In one optional embodiment of this specification, the initial inference trajectory includes at least one candidate knowledge gap, and at least one task includes a search decision gating task; Using the initial inference model, at least one task inference is performed based on pre-trained prompts to obtain the initial inference trajectory, including: The search decision gating task is performed using a pre-trained inference model. At least one candidate knowledge gap is obtained based on the pre-trained prompt words. If the influence of at least one candidate knowledge gap is less than a preset influence threshold, at least one candidate knowledge gap is abandoned. An initial inference trajectory is generated based on the pre-trained prompts and the execution results of the search decision gating task.

[0157] A preset impact threshold is a threshold value used to determine whether a candidate knowledge gap is worth triggering subsequent retrieval operations. Specifically, the preset impact threshold can be a pre-set fixed value or an adaptive threshold dynamically calculated based on the length, complexity, or historical statistical data of the current pre-trained prompts. The preset impact threshold can be used to compare the impact of candidate knowledge gaps. When the impact is below the threshold, it indicates that the gap has a small impact on the image generation quality, and even without retrieval supplementation, it will not lead to significant generation errors. Furthermore, setting the preset impact threshold can balance retrieval benefits and costs. A threshold that is too low will lead to a large number of low-impact gaps triggering searches, increasing noise injection and inference overhead; a threshold that is too high may miss key gaps that, while having moderate impact, genuinely require external knowledge.

[0158] Abandonment is a decision-making behavior that neither triggers subsequent retrieval operations for candidate knowledge gaps nor includes them in the target knowledge gap set. Specifically, abandonment can occur in search decision gating tasks. When a pre-trained inference model determines that the influence of one or more candidate knowledge gaps is less than a preset influence threshold, the model chooses to ignore these gaps, not generate retrieval indexes for them, and not perform the steps related to the gap in the reference filtering and reference integration tasks. Abandonment can be used to prevent concepts that the generator already possesses accurate knowledge of or that have minimal influence from being included in the retrieval process, thereby avoiding concept erosion and unnecessary noise injection.

[0159] The execution result of the search decision gating task is a set of decision information output after the gating task is completed. This includes a list of identified candidate knowledge gaps, an impact assessment value for each gap, and a decision label indicating whether each gap is abandoned or selected as the target knowledge gap. Specifically, the execution result may include a decision vector, where each element corresponds to a candidate knowledge gap and has a value of "abandon" or "accept". The execution result can be used as part of the initial inference trajectory, ensuring that the trajectory includes not only the final target knowledge gap but also all decision information from the gating process, including the gaps that were abandoned and the reasons for their abandonment. Furthermore, the execution result may also include intermediate calculations comparing the model's impact on each candidate knowledge gap with a preset impact threshold.

[0160] This specification's embodiments introduce a preset influence threshold and perform an abandonment operation when the influence of candidate knowledge gaps is less than this threshold. This allows the pre-trained inference model to actively filter out knowledge gaps with minimal impact on image generation quality during the gating stage. This abandonment mechanism directly prevents concept erosion: for concepts that the generator already possesses accurate knowledge of or have extremely low influence, even if the model identifies them as candidate knowledge gaps, it will abandon triggering the search because the influence is insufficient, thus avoiding unnecessary external references interfering with the generator's existing knowledge. Simultaneously, the model learns not only to adopt gaps with high influence but also to abandon gaps with low influence, enabling the pre-trained inference model to make more accurate gating decisions when faced with unseen prompts, minimizing unnecessary retrieval counts and noise injection without sacrificing generation quality.

[0161] An optional embodiment of this specification describes the training steps for a target multimodal generative model, including: Obtain sample-generated prompts, an initial multimodal generation model, and a pre-trained inference model; Using a pre-trained inference model, inference is performed based on sample-generated prompts to obtain the pre-trained inference trajectory; Using an initial multimodal generative model, multiple sampled candidate images are generated based on pre-trained inference trajectories; Each sampled candidate image is evaluated and scored according to the evaluation protocol to obtain an evaluation score for each sampled candidate image; Based on the evaluation scores, image preference pairs are determined from each sampled candidate image; Based on image preference pairs, the initial multimodal generative model is trained with preference optimization until the target multimodal generative model is obtained.

[0162] Sample generation prompts are textual information describing the content of the desired generated image, used during the preference optimization training phase of the initial multimodal generative model. Their role is to trigger the pre-trained inference model to generate corresponding pre-trained inference trajectories. Specifically, sample generation prompts can be a short phrase describing a specific scene, object, person, or style, or a complete sentence containing multiple entities and their attribute relationships. Sample generation prompts can be used as input to the pre-trained inference model to drive the model to perform inference tasks such as knowledge gap identification, retrieval decision-making, reference filtering, and integration, ultimately outputting target prompts with supplemented knowledge. Furthermore, sample generation prompts can share the same data source as pre-trained prompts and can be constructed independently.

[0163] The initial multimodal generative model is a multimodal generative model that has not undergone preference optimization training or has only undergone basic training. Its parameters need to be updated through preference optimization training to obtain the target multimodal generative model. The initial multimodal generative model can be a diffusion model, a generative adversarial network, or a vector quantization-based generative model, and its initial parameters can be randomly initialized or loaded from publicly available pre-trained weights. The initial multimodal generative model can be used to receive target prompts from pre-trained inference trajectories and sample to generate multiple sampled candidate images. During preference optimization training, the initial multimodal generative model can gradually improve its ability to generate long-tail knowledge by comparing positive and negative preference images.

[0164] A pre-trained inference trajectory is a structured data sequence output by a pre-trained inference model after a single complete inference iteration on a sample prompt word. This trajectory can include gating decision results, retrieval queries, selected reference information, and adjusted target prompt words. Specifically, the pre-trained inference trajectory may include identified target knowledge gaps, image information of the target reference image retrieved based on the target knowledge gaps, and the target prompt words obtained after adjusting the original sample prompt words. The pre-trained inference trajectory can be used as conditional input when the initial multimodal generative model generates sampled candidate images, guiding the generator to generate images based on supplemented external knowledge. Furthermore, for the same sample prompt word, the pre-trained inference model can generate multiple different pre-trained inference trajectories, each corresponding to a different search decision path.

[0165] Sampling candidate images are multiple images generated by the initial multimodal generative model through multiple random samplings based on conditional information in the pre-trained inference trajectory. Specifically, sampling candidate images can be different image variants obtained under the same conditional input by changing the random seed in the diffusion model or the latent noise vector in the generative adversarial network. Sampling candidate images can be used as the scoring objects of subsequent evaluation protocols. By comparing the quality differences of different sampling candidate images under the same pre-trained inference trajectory, preference pairs for preference optimization training are constructed.

[0166] Evaluation protocol scoring is a process of quantitatively evaluating sampled candidate images based on preset evaluation dimensions and scoring rules. This process can employ a visual language model as an automatic judge. Specifically, evaluation protocol scoring can score each sampled candidate image in multiple dimensions according to the pre-annotated list and gauging in the sample prompts, or it can use general image quality assessment metrics. Evaluation protocol scoring can generate an evaluation score for each sampled candidate image, which reflects the quality level of the image under given input conditions. Furthermore, the dimensions of evaluation protocol scoring can include list compliance, gauging score, prompt follow-through, text rendering effect, image artifacts, visual reference fidelity, and text reference fidelity, etc.

[0167] The evaluation score is a numerical quality assessment result output by the evaluation protocol scoring process for each sampled candidate image. Specifically, the evaluation score can be any numerical value, with higher values ​​indicating better image quality and a higher degree of alignment with cue words and reference information. The evaluation score can be used to rank and compare multiple sampled candidate images corresponding to the same pre-trained inference trajectory, thereby distinguishing images of relative quality. Furthermore, when the evaluation protocol includes multiple dimensions, the evaluation score can be a weighted sum of multiple evaluation dimensions or the average of the scores for each dimension.

[0168] Image preference pairs are paired data consisting of a positive preference image and a negative preference image selected from multiple sampled candidate images corresponding to the same pre-trained inference trajectory, based on their evaluation scores. Specifically, the positive preference image can be the sampled candidate image with the highest evaluation score, and the negative preference image can be the sampled candidate image with the lowest evaluation score. Image preference pairs can be used as supervision signals in preference optimization training, instructing the generator to tend to produce outputs similar to positive preference images and reduce the production of outputs similar to negative preference images under the same input conditions. Furthermore, one pre-trained inference trajectory can generate one image preference pair, and multiple pre-trained inference trajectories can generate multiple image preference pairs, which together constitute the dataset for preference optimization training.

[0169] Preference optimization training is a training method that directly optimizes generator parameters based on image preferences. Specifically, preference optimization training can employ a direct preference optimization algorithm to update generator parameters by maximizing the preference probability of positively preferred images relative to negatively preferred images. Preference optimization training can be used to enable an initial multimodal generative model to learn the distribution characteristics of generating high-quality images under given input conditions, thereby internalizing visual knowledge that would otherwise require external retrieval into the model parameters. Furthermore, preference optimization training can be conducted online iteratively, expanding the generator's knowledge boundary after each training round, thus allowing for collaborative calibration with the inference model.

[0170] In practical applications, there are various ways to optimize the training of the initial multimodal generative model based on image preferences.

[0171] An alternative approach employs a direct preference optimization algorithm: for each image preference pair containing a positive preference image and a negative preference image, a preference contrast loss function is constructed. This loss function encourages the generator to increase the probability of generating a positive preference image while decreasing the probability of generating a negative preference image under the same input conditions. The loss function is typically defined based on the generator's implicit reward model. By minimizing this loss, the generator's parameters are progressively updated, shifting its distribution towards the positive preference image.

[0172] Another alternative approach is to employ iterative online preference optimization: in each training round, new candidate images are generated using the current generator, and an evaluation protocol is immediately applied to score and construct preference pairs. These newly constructed preference pairs are then used to update the generator parameters; the updated generator then enters the next round of sampling and training. This process iterates until the generator's generation quality converges or a preset number of iterations is reached. This online approach allows the generator to continuously improve itself, gradually internalizing more external knowledge.

[0173] The preference optimization training formula can be referenced as follows:

[0174] in, Sample prompts (conditions); The "winning" image (positive preference image) in the preference pair; The "failure" image (negative preference image) in the preference pair; The current generative model to be trained (parameter θ); Generate a reference model (usually an initial model with parameters frozen before training begins). These are hyperparameters that control the degree of deviation from the reference model; This is the sigmoid function.

[0175] In this scheme, the loss function is used to perform preference optimization training on the target multimodal generative model. The image preference pairs in the training data ( , (Source: from the same prompt word) After multiple candidate images are evaluated and scored by an evaluation protocol, a pair of images is selected based on their scores (the image with a positive preference is the high-scoring image, and the image with a negative preference is the low-scoring image).

[0176] This embodiment acquires sample prompts, an initial multimodal generation model, and a pre-trained inference model. The pre-trained inference model uses the sample prompts to infer a pre-trained inference trajectory. The initial multimodal generation model then samples and generates multiple candidate images based on this inference trajectory. An evaluation protocol is used to score each image, quantifying the differences in generation quality under different input conditions. Image preference pairs are determined from the candidate images based on the evaluation scores, effectively constructing a comparison of "good" and "bad" generation under the same input conditions. This comparison signal directly reflects the quality differences within and outside the generator's current capability boundary. Finally, the initial multimodal generation model is trained with preference optimization based on image preferences. This allows the generator to gradually strengthen its behavior pattern of generating positively preferred images and suppress its behavior pattern of generating negatively preferred images through contrastive learning, thereby gradually internalizing long-tail visual knowledge that originally required external retrieval into the generator's parameters. The target multimodal generation model obtained through this training has a significantly expanded knowledge boundary compared to the initial model.

[0177] In one optional embodiment of this specification, the image preference pair includes a positive preference image and a negative preference image; Based on the evaluation scores, image preference pairs are determined from each sampled candidate image, including: Based on the evaluation scores, positive preference images and negative preference images are determined from each sampled candidate image; Based on image preference pairs, the initial multimodal generative model is trained with preference optimization until the target multimodal generative model is obtained, including: Construct a preference contrast loss based on positive and negative preference images; Based on preference contrast loss, the initial multimodal generative model is trained with preference optimization until the target multimodal generative model is obtained.

[0178] A positive preference image is an image selected based on its evaluation score from multiple sampled candidate images corresponding to the same pre-trained inference trajectory, choosing the one whose quality matches the desired generated result. Specifically, the positive preference image can be the image with the highest evaluation score among multiple candidate images generated by multiple random samplings. The positive preference image can be used as a positive sample in preference optimization training, indicating that the generator should be more inclined to produce outputs similar to that image under the same conditions.

[0179] A negative preference image is the image whose quality is least in line with the desired generated result, selected based on its evaluation score from multiple sampled candidate images corresponding to the same pre-trained inference trajectory. Specifically, the negative preference image can be the image with the lowest evaluation score among multiple candidate images generated by multiple random samplings. The negative preference image can be used as a negative sample in preference optimization training, instructing the generator to reduce the generation of outputs similar to that image under the same conditions.

[0180] Preference contrastive loss is a loss function built upon positive and negative preference images, used to guide the update direction of generator parameters during preference optimization training. Specifically, preference contrastive loss can be a loss function based on the generator's implicit reward model, which encourages the generator to increase the probability of generating positive preference images while decreasing the probability of generating negative preference images given the same input conditions. Preference contrastive loss can replace the two-stage approach in traditional reinforcement learning that requires separate training of the reward model, achieving end-to-end direct preference optimization.

[0181] In practical applications, there are multiple ways to determine positive and negative preference images based on evaluation scores.

[0182] One alternative approach is to compare the evaluation scores of each sampled candidate image for each pre-trained inference trajectory with multiple sampled candidate images, select the one with the highest score as the positive preference image, and select the one with the lowest score as the negative preference image, thereby maximizing the quality difference between positive and negative samples and making the preference contrast loss have a stronger discriminative signal.

[0183] Another alternative approach is to set a score threshold, and only build preference pairs if the difference between the highest and lowest scores exceeds that threshold.

[0184] Another option is to select the average of the top K sampled candidate images with the highest scores or randomly select one of them as the positive preference image, and to select the average of the bottom K images with the lowest scores or randomly select one of them as the negative preference image.

[0185] This specification's embodiments determine positive and negative preference images from each sampled candidate image based on evaluation scores. This automatically constructs paired samples with a clear contrast in quality under the same input conditions, avoiding the high cost of manually labeling preference data. The positive preference image represents the top-ranked results the generator can produce given external knowledge supplementation, while the negative preference image represents lower-quality failures; together, they constitute a natural comparative learning signal. A preference contrast loss is constructed based on the positive and negative preference images. This loss function directly utilizes the generator's implicit reward distribution, updating the generator parameters by maximizing the preference probability of the positive preference image relative to the negative preference image. The initial multimodal generation model is trained using this preference contrast loss. The generator gradually learns to avoid decision paths leading to low quality under the same input conditions, tending towards producing high-quality output. As training progresses, the generator gradually internalizes visual knowledge that previously required external retrieval into its parameters, effectively expanding its knowledge boundaries. This allows for accurate generation of similar or identical knowledge gaps in subsequent applications without relying on retrieval.

[0186] One optional embodiment of this specification involves evaluating the advantage of each candidate image sample to obtain a relative advantage value for each candidate image sample, including: Based on a pre-trained advantage evaluation model, the advantage of each candidate image sample is evaluated in a preset evaluation dimension to obtain the relative advantage value of each candidate image sample. The preset evaluation dimension includes at least one of the following: checklist compliance, rubric score, cue word follow-through, text rendering effect, image artifacts, visual reference fidelity, and text reference fidelity.

[0187] A pre-trained advantage assessment model is a visual language model or discriminative model that has been pre-trained to perform multi-dimensional quality assessment of images. Specifically, a pre-trained advantage assessment model can take an image as input and output the image's score on various preset evaluation dimensions. Pre-trained advantage assessment models can be used to replace human judges to perform automated and large-scale advantage assessments of candidate images, thereby obtaining the relative advantage value of each candidate image.

[0188] Preset evaluation dimensions are a set of scoring items pre-defined during the advantage assessment process to measure image quality from different perspectives. Specifically, preset evaluation dimensions may include one or more sub-dimensions related to image generation quality, each corresponding to a specific quality focus, such as the alignment of image content with prompts, image realism, accuracy of text rendering, and fidelity of reference information. Preset evaluation dimensions can guide pre-trained advantage assessment models to comprehensively evaluate image quality from multiple perspectives, enabling relative advantage values ​​to more comprehensively reflect the quality of candidate images and avoid bias from single-dimensional evaluation. Furthermore, different combinations of preset evaluation dimensions can be used for different application scenarios; for example, in image tasks requiring text generation, the weight of text rendering quality can be increased accordingly.

[0189] Checklist compliance measures the degree to which a candidate image contains the key visual elements labeled in the sample cue words, typically expressed in binary or hierarchical form. Specifically, checklist compliance checks each item in the candidate image against a pre-labeled list of visual elements and calculates the presentation ratio or a weighted score. Checklist compliance can be used to assess how well the generated image adheres to the key content explicitly specified by the user; a high checklist compliance indicates that the generator has not omitted important visual information explicitly requested by the user.

[0190] Gauge scoring is a quality score for candidate images across a predefined failure category dimension, used to measure whether an image contains specific types of errors or defects. Specifically, gauge scoring can include multiple predefined failure categories, such as structural distortion, color errors, scale inconsistencies, and texture anomalies. Gauge scoring can be used to capture generation quality issues unrelated to cue word entities that cannot be covered by checklist compliance, such as overall visual coherence and lighting appropriateness of the image.

[0191] Cue word following measure is a comprehensive semantic alignment metric that assesses how well a candidate image as a whole follows the original cue words and their supplementary target cue words. Specifically, cue word following can be calculated based on cross-modal similarity between the image and the cue words using a visual language model, or it can be represented by a comprehensive alignment score output by the model through pre-training advantages. Cue word following can be used to evaluate whether the generator, after integrating external reference information, can fully reflect the requirements of the cue words in the image. Furthermore, cue word following differs from checklist compliance in that checklist compliance only focuses on pre-annotated key visual elements, while cue word following covers all semantic content within the cue words.

[0192] Text rendering quality measures the clarity, accuracy, and style matching of text regions appearing in candidate images. Specifically, text rendering quality includes text readability, spelling accuracy, font style matching, and the rationality of the text's position and layout within the image. Text rendering quality can be used to evaluate a generator's ability to handle text-based visual elements, as text rendering is a common weakness in current image generation models. Furthermore, the score for text rendering quality can be obtained by using an optical character recognition model to detect text regions in an image and compare them with the target text.

[0193] Image artifactimetry is a visual quality dimension that measures whether a candidate image appears unrealistic, unnatural, or "fake," and is used to evaluate the realism of an image. Specifically, image artifactimetry can reflect the presence of unnatural phenomena such as unreasonable shadows, reflections, texture repetition, abnormal object edges, and disproportionate limbs. It can also reflect whether the overall image possesses visual characteristics similar to a real photograph or high-quality artistic creation. Image artifactimetry can be used to evaluate the generator's performance in terms of visual realism; low artifactimetry indicates that the image looks natural and realistic, while high artifactimetry indicates obvious synthetic traces or visual artifacts. Furthermore, the image artifactimetry score can be given by a pre-trained advantage evaluation model by analyzing the low-level statistical features and high-level semantic consistency of the image.

[0194] Visual reference fidelity measures the degree to which the visual components of a candidate image accurately retain the key attributes of the target reference image. Specifically, visual reference fidelity checks the accuracy of the reproduction of each attribute in the candidate image, labeled as needed in the target reference image, for example, by calculating color histogram similarity or using a visual language model to determine attribute consistency. High fidelity indicates that the generator can accurately extract and apply the visual signals from the reference, while low fidelity indicates that reference information has been lost or distorted during generation.

[0195] Text reference fidelity measures the degree to which the visual components of a candidate image, related to the textual description in the target reference image, accurately follow that textual description. Specifically, text reference fidelity checks whether the corresponding visual attributes in the candidate image match the textual description contained in the target reference image. A visual language model can be used to evaluate the degree of matching between the image and the textual description. High fidelity indicates that the generator can correctly convert semantic descriptions into visual attributes. Furthermore, text reference fidelity and visual reference fidelity can each correspond to two different modalities of reference input, together forming a complete reference fidelity evaluation system.

[0196] This specification's embodiments utilize a pre-trained advantage evaluation model to assess the advantages of each candidate image sample across preset evaluation dimensions, enabling automated, multi-dimensional image quality quantification. Specifically, the preset evaluation dimensions include at least one of the following: checklist compliance, rubric score, cue word adherence, text rendering effect, image artifactiness, visual reference fidelity, and text reference fidelity. These dimensions comprehensively cover the quality evaluation requirements of generated images from multiple perspectives, such as content completeness, visual realism, semantic alignment, and reference information adherence, offering higher granularity and interpretability compared to traditional single-score or manual scoring. The pre-trained advantage evaluation model automatically outputs scores for these dimensions and calculates the relative advantage value of each candidate image sample based on these scores. This allows generated images corresponding to different inference trajectories under the same cue word to be compared under a unified multi-dimensional standard, thereby accurately identifying which inference trajectory produced a better generation effect. This relative advantage value provides a quantitative and objective basis for selecting positive advantage inference trajectories from multiple sample inference trajectories, avoiding the subjectivity of manual selection, and enabling the entire training process to be executed automatically and on a large scale. It also provides a reliable source of supervision signals for the rejection sampling fine-tuning of the inference model.

[0197] One optional embodiment of this specification involves obtaining sample prompt words, including: Get the initial prompt word; Perform semantic analysis on the initial prompt words to obtain the prompt word entities; Based on each prompt word entity, determine the label image corresponding to each prompt word entity; Using a pre-defined prompt word generation model, sample prompt words are generated based on the label images corresponding to each prompt word entity and each boosting word entity.

[0198] Initial prompts are raw text inputs that have not been processed by the sample prompt generation process, serving as the foundational material for constructing sample prompts. Specifically, initial prompts can be raw text collected from real user requests, text portions of image-text pairs extracted from public datasets, or simple descriptive statements automatically generated using templates. Initial prompts can contain one or more visual concepts, entity names, and their attribute descriptions, but they may not themselves possess complete labeled images or structured entity annotations. Furthermore, initial prompts can cover general objects and long-tail entities, enabling subsequent processes to generate sample prompts covering different knowledge types.

[0199] Cue word entities are visual concept units with independent semantic meaning extracted from initial cue words through semantic analysis. Examples include person names, object categories, cultural symbols, and specific attributes. Specifically, a cue word entity can be a noun phrase, a proper noun, or a descriptive phrase with modifiers. Cue word entities can be used as anchor points to determine the label image; each entity corresponds to a specific visual content to be generated or retrieved. Furthermore, an initial cue word can yield multiple cue word entities, which may have combination or modification relationships, such as a main entity and its attribute entities.

[0200] A label image is a reference image corresponding to a cue word entity and used to characterize the standard visual appearance of that entity. Specifically, a label image can be a high-quality image retrieved from a public image library, a standard schematic diagram pre-drawn by a human designer, or a sample image pre-synthesized by other generative models. Label images can be used as visual anchors when generating sample cue words, enabling the final generated sample cue words to be associated with the actual visual features of the entity.

[0201] A pre-trained prompt word generation model is a generative model that generates or assists in generating structured sample prompt words based on prompt word entities and their corresponding label images. Specifically, the pre-trained prompt word generation model can be a multimodal language model, taking a list of prompt word entities and their corresponding label images as input, and outputting a natural language descriptive text as sample prompt words. The pre-trained prompt word generation model can be used to transform discrete entities and visual references into coherent prompt word text that conforms to natural language habits, thereby automatically constructing large-scale training corpora. Furthermore, the pre-trained prompt word generation model can be implemented using various methods, such as template filling, retrieval-based rewriting, or generation based on large language models.

[0202] It should be noted that in practical applications, the pre-trained prompts and sample-generated prompts may use the same batch of prompts as the sample prompts, or they may use different prompts. In addition, the construction process of the pre-trained prompts and sample-generated prompts may be the same as the construction process of the sample prompts.

[0203] In practical applications, there are several ways to generate sample prompts using a preset prompt word generation model.

[0204] An alternative approach is to use a template-based generation method for the preset prompt word generation model: design a set of natural language templates in advance, then fill the prompt word entities into the corresponding positions in the templates according to their semantic roles, and use the preset prompt word generation model to help refine them, finally forming sample prompt words.

[0205] Another alternative approach is to retrieve prompt words from an existing high-quality prompt word library that are more relevant to the current entity set than a preset relevance threshold as prototypes. Then, a language model is used to replace the entity names in the prototypes with the names of the current entities, and modifiers are adjusted to match the visual details in the label images.

[0206] Another option is to use a multimodal large language model as the pre-defined prompt word generation model: Each prompt word entity and its corresponding label image are taken as multimodal input, allowing the model to understand the visual appearance of the entities and output a natural language description as sample prompt words. The model can automatically generate reasonable descriptions of relationships between entities, their positional layout, and style modifications, making the generated sample prompt words closer to the natural language distribution of real user requests.

[0207] This specification describes an embodiment that obtains initial prompt words and performs semantic analysis to obtain prompt word entities. This decomposes the original natural language text into controllable and labelable discrete semantic units, allowing for the independent determination of a label image for each entity. A preset prompt word generation model generates sample prompt words based on each prompt word entity and its corresponding label image, achieving automatic conversion from discrete entities and visual references to coherent natural language text. The resulting sample prompt words have the following characteristics: because each entity is associated with a label image, the generated sample prompt words naturally carry a visual checklist that can be used for automatic evaluation, providing a basic annotation for subsequent advantage assessment and evaluation protocol scoring.

[0208] In one optional embodiment of this specification, before generating sample prompts based on the label images corresponding to each prompt entity and each raised word entity using a preset prompt generation model, the method further includes: The proportion of each entity in the sample set is obtained by statistically analyzing the entities of the labeled prompt words. Based on the proportion of each entity in the sample set, construct the estimated training set frequency of the long-tail distribution; Entities are filtered by estimating the training set frequency using the long-tail distribution, and the filtered prompt word entities are obtained.

[0209] Annotated cue words are visual conceptual units with clear semantic meanings, obtained manually or automatically from existing datasets, and are usually associated with specific image content. Annotated cue words can be used as raw data for statistical sample set distributions, reflecting the prevalence of the visual concept in the real world by calculating the frequency of each entity's occurrence in the sample set. Furthermore, annotated cue words can cover a wide distribution, ranging from high-frequency common objects to extremely low-frequency long-tail entities.

[0210] A sample set is a dataset used to statistically analyze the distribution of entity prompts. It typically consists of a large number of real user prompts or text collected from publicly available data sources. The sample set can reflect the frequency distribution of various entities in user requests within real-world application scenarios. This distribution usually exhibits long-tail characteristics, with a few high-frequency entities accounting for the majority of occurrences, while a large number of low-frequency entities appear only a very small number of times.

[0211] The proportion of each entity in the sample set is the ratio of the frequency of each labeled prompt word entity to the total frequency of all entities in the sample set, used to quantify the commonness of that entity. Specifically, for an entity, its proportion equals the number of times that entity appears in the sample set divided by the sum of the frequencies of all entities. The proportion can be used to distinguish between high-frequency entities and long-tail entities. Furthermore, the distribution pattern of the proportion determines the sampling weights when constructing the long-tail distribution training set.

[0212] The estimated training set frequency of the long-tail distribution is a frequency distribution used to guide the sampling probability of entities when generating sample prompts, obtained by transforming or adjusting the original proportions of each entity in the sample set. Specifically, the long-tail distribution can be a mathematical transformation of the original proportions, or it can be achieved by setting a threshold to divide entities into high-frequency heads and low-frequency tails and assigning different sampling weights to each. The estimated training set frequency of the long-tail distribution can be used to weaken the dominance of high-frequency entities in the generated sample prompt set and enhance the representativeness of low-frequency long-tail entities, thereby simulating the visual knowledge bottleneck of deep long tails in real user requests.

[0213] The filtered prompt entities are a subset of entities obtained by filtering the original prompt entities based on the estimated training set frequencies of the long-tail distribution. These entities serve as input to the preset prompt generation model to generate sample prompts. Specifically, the filtered prompt entities can include those entities whose sampling probability in the estimated training set frequencies is higher than a certain threshold. The filtered prompt entities can be used to control the distribution of the final generated sample prompt set, ensuring it conforms to the expected long-tail characteristics. Furthermore, the proportion of long-tail entities in the filtered prompt entities is significantly higher than their original proportion in the real sample set, thus making the trained inference model and generator more attentive to and adept at handling long-tail visual knowledge.

[0214] This specification's embodiments obtain the proportion of each entity in the sample set by statistically analyzing the labeled prompt word entities. This quantifies the prevalence of various visual concepts in real user requests, revealing typical long-tail distribution characteristics: a few entities appear frequently while a large number of entities appear rarely. The estimated training set frequency of the long-tail distribution is constructed based on the proportion of each entity in the sample set. This transforms the original proportions, increasing the relative probability of low-frequency entities, which originally had a very low proportion in the sample set, in the estimated training set frequency. The estimated training set frequency of the long-tail distribution is used to filter entities, resulting in filtered prompt word entities. This enhances the representativeness of low-frequency long-tail entities in the filtered entity set, while appropriately weakening the dominance of high-frequency entities. The filtered prompt word entities are used for subsequent sample prompt word generation. The final training corpus is actively biased towards the long tail in distribution, realistically simulating the deep long-tail characteristics of user requests in actual application scenarios. Compared to directly using the training set constructed from the original distribution, the sample prompt words obtained through this long-tail enhancement filtering allow the subsequently trained inference and generation models to encounter more rare but crucial visual concepts during training, thereby improving the model's generalization ability in real-world long-tail scenarios.

[0215] Figure 2 A flowchart illustrating the processing steps of a model training method provided in one embodiment of this specification includes: By using a collaborative training mechanism to calibrate the inference engine and the image generator, the problem of generating long-tail visual knowledge can be solved.

[0216] Specifically, after the user inputs a prompt text, the prompt is fed into the Visual Language Model Inferencer (VLM Inferencer). The inferencer first performs semantic analysis on the user prompt, identifying any knowledge gaps—visual concepts that exceed the parameterized knowledge range of the image generator, such as details of a specific character's appearance, regional cultural symbols, or niche font styles. For each identified knowledge gap, the inferencer assesses its impact on image generation, triggering an external retrieval operation only when the impact exceeds a preset threshold. The retrieval operation can select either image search or text search modality based on the type of knowledge gap to obtain candidate reference information. The inferencer then filters the retrieval results based on the principle of information complementarity, selecting reference information that can fill specific gaps and contains the least amount of irrelevant content, avoiding references with excessively high overall matching to prevent copying effects. Next, the inferencer integrates the original user prompt, the identified knowledge gap analysis, and the filtered reference information into an enhanced reference prompt term. This reference prompt term conveys the visual attributes of the reference image through natural language description and includes explicit or implicit usage guidance, instructing the image generator which attributes need to be strictly borrowed and which parts can be freely adapted.

[0217] In the collaborative training framework, the scheme comprises two complementary phases. The first phase teaches the image generator what it can internalize. Specifically, a pre-trained inferencer provides search-enhanced text specifications for training prompts, enabling the image generator to access previously lacking external knowledge. The image generator is then trained using an online iterative Direct Preference Optimization (DPO) method: in each training epoch, the image generator samples multiple candidate images for the same enhanced prompt, scores each image using an evaluation protocol, and constructs preference pairs from the highest and lowest-scoring generated results. The DPO loss function encourages the generator to increase the probability of generating high-scoring images and decrease the probability of generating low-scoring images, thereby gradually internalizing knowledge with sufficiently stable visual identities that can be absorbed into the model weights. After this training phase, the image generator's knowledge boundary is expanded; some concepts that previously required retrieval for correct generation can now be accurately generated directly by the generator using parameterized knowledge.

[0218] The second stage is used to learn content that the search generator cannot be taught. With the training in the first stage, the knowledge boundary of the image generator has shifted, while the inferencer's search strategy still uses the original boundary, causing the inferencer to perform unnecessary searches for concepts already internalized by the generator. To address this, the scheme employs rejection sampling fine-tuning to recalibrate the inferencer. Specifically, the updated image generator obtained from the first stage training is paired with the initial inferencer. For each training prompt, multiple inference trajectories are developed, each containing gating decisions, retrieval operations, and integration results. The updated image generator generates corresponding images based on the reference prompts for each trajectory, and scores the images using an evaluation protocol. Among the multiple trajectories for the same prompt, the relative advantage value of the generated images is calculated for each trajectory. Only trajectories with positive relative advantage values ​​are retained; these trajectories where the inferencer correctly abandons unnecessary searches receive high scores and are reinforced, while trajectories that trigger unnecessary searches leading to image quality degradation receive low scores and are discarded. Then, the inferencer is trained using supervised fine-tuning based on these positive advantage trajectories. In this way, the inference engine naturally learns the knowledge boundaries of the enhanced image generator from the reward signal without explicit boundary labels: for concepts that the generator can internalize, the inference engine learns to give up searching; for concepts that the generator still cannot accurately generate, the inference engine learns to correctly trigger searches and integrate reference information.

[0219] In the actual inference phase, after the user inputs the prompt word to be generated, the target inference model, obtained through the aforementioned collaborative training, first performs a gating judgment. If it is determined that there is no knowledge gap worth retrieving, the original prompt word is directly sent to the target image generation model as a reference prompt word; if it is determined that there is a target knowledge gap that needs to be retrieved, the inference model sequentially performs reference filtering and reference integration tasks to generate reference prompt words containing external knowledge supplementation. Based on these reference prompt words, the target image generation model utilizes its internalized parameterized knowledge and semantic guidance from the reference prompt words to ultimately generate a high-quality image that contains accurate long-tail visual details while retaining creative composition. Through this two-stage collaborative training, the search strategy of the inferencer and the actual capabilities of the image generator are always matched, effectively breaking through the fixed knowledge boundaries of the image generator while avoiding concept erosion and copying effects, and significantly improving the generation effect for deep long-tail user requests.

[0220] The following is in conjunction with the appendix Figure 3 This paper will further illustrate the application of the model training method provided in this manual to content reasoning scenarios, using it as an example. Figure 3 A flowchart of a content reasoning method provided in one embodiment of this specification specifically includes the following steps: Step 302: Input the initial content into the target inference model, execute the search gating task, obtain multiple candidate knowledge gaps corresponding to the initial content, and determine the target knowledge gap corresponding to the initial content from the candidate knowledge gaps.

[0221] The initial content refers to the raw data input into the target inference model to trigger the multimodal content generation process. Specifically, the initial content can be user-inputted text prompts, pre-processed text descriptions, or multimodal input data containing text and reference image information. The initial content can serve as the foundational input for the target inference model to perform search gating, reference filtering, and reference integration tasks. Furthermore, the initial content may contain one or more concepts or entities that require external knowledge supplementation; these are identified as candidate knowledge gaps in subsequent gating tasks.

[0222] A target reasoning model is a reasoning model obtained through model training methods that can adaptively make search decisions on input content and output enhanced reasoning content. Specifically, the target reasoning model can be a visual-language model that has undergone supervised pre-training and rejection sampling fine-tuning, internally learning gating, filtering, and integration strategies that match the current knowledge boundaries of the target multimodal generative model. In practical applications, the target reasoning model can receive initial content, sequentially perform search gating, reference filtering, and reference integration tasks, and output enhanced reasoning content supplemented with knowledge. Furthermore, the target reasoning model can automatically determine when to trigger retrieval and when to abandon retrieval, and integrate the retrieved reference information into the reasoning content through natural language descriptions, thereby avoiding concept erosion and copying effects.

[0223] Candidate knowledge gaps are sets of potential concepts identified by the target inference model after semantic analysis of the initial content, which may exceed the parameterized knowledge scope of the target multimodal generative model. Specifically, candidate knowledge gaps may include specific entities involved in the initial content (such as character names), attributes of those entities (such as clothing color and texture), spatial or relational constraints between entities, etc. Each candidate knowledge gap is associated with an influence evaluation value. Candidate knowledge gaps can be used as screening objects for search gating tasks, and only a portion of these gaps will be identified as target knowledge gaps. Furthermore, the identification of candidate knowledge gaps does not depend on the annotation information during the training phase, but is entirely determined by the parameterized knowledge of the target inference model.

[0224] Target knowledge gaps are key knowledge concepts selected from candidate knowledge gaps that are deemed worthy of triggering actual retrieval operations. Specifically, target knowledge gaps are those concepts whose influence exceeds a preset threshold and which the target multimodal generation model cannot accurately generate or may generate errors in. Target knowledge gaps can be used as the basis for generating retrieval queries in reference filtering tasks, driving external retrieval systems to return relevant reference content. Furthermore, since the target inference model has been co-trained with the target multimodal generation model, the target knowledge gaps it selects are highly matched with the actual knowledge boundaries of the generator, enabling it to abandon searches when unnecessary retrieval is required.

[0225] Regarding how to utilize the target inference model to perform the search gating task, multiple candidate knowledge gaps are obtained based on the initial content, and the target knowledge gap is determined from these candidate knowledge gaps: In practical applications, there are multiple ways to perform search gating tasks and identify target knowledge gaps.

[0226] In one alternative approach, the target inference model first performs full-text semantic analysis on the initial content, identifying all entities and key attributes as candidate knowledge gaps, and outputs an influence score for each candidate knowledge gap. Then, the model identifies candidate knowledge gaps whose influence exceeds a preset threshold as target knowledge gaps. This threshold can be a fixed value or dynamically adjusted based on the complexity of the initial content.

[0227] Alternatively, the target inference model can simultaneously output the type label and influence score for each candidate knowledge gap, and then determine the target knowledge gap according to a predefined priority rule. For example, visual identity gaps often have a higher priority, and may be prioritized even if their influence is slightly lower than other types of gaps.

[0228] Another option is that the target reasoning model can use multi-round gating decision-making. First, it initially screens out several candidate knowledge gaps with high impact. Then, it generates a trial retrieval query for each candidate gap and performs a fast retrieval. Based on whether there is high-quality reference information in the fast retrieval results, it finally decides whether to upgrade the gap into a target knowledge gap.

[0229] Step 304: Utilize the target inference model to perform a reference filtering task. Based on the target knowledge gap corresponding to the initial content, retrieve the image information of multiple candidate reference images corresponding to the initial content, and determine the image information of the target reference image corresponding to the initial content from the image information of the multiple candidate reference images corresponding to the initial content.

[0230] The image information of candidate reference images consists of multiple reference contents and their associated metadata returned from an external retrieval interface that are related to the target knowledge gap. Specifically, the image information of candidate reference images may include image pixel data, image thumbnails, source URLs, similarity scores, and relevant descriptive information extracted from web page text. This information may contain valid signals directly related to the target knowledge gap, as well as information noise unrelated to the gap. The image information of candidate reference images can be used as input for a reference filtering task, from which the target inference model needs to select the most suitable reference content. Furthermore, the relevance of the candidate reference information directly determines the effectiveness of the integration task.

[0231] The image information of the target reference image is selected from the image information of candidate reference images based on the principle of information complementarity, and is used to fill the target knowledge gap. Specifically, the image information of the target reference image should only contain visual attributes directly related to the target knowledge gap, while discarding background, pose, or other object information unrelated to the gap. The image information of the target reference image can be used as an information source for the reference integration task. The target inference model converts the key attributes into natural language descriptions and integrates them into the inference content. Furthermore, the selection objective of the target reference image is to minimize the copy effect, avoiding the selection of references with excessively high overall matching that may induce the generator to directly copy.

[0232] In practical applications, there are multiple ways to perform reference filtering tasks and determine the image information of the target reference image.

[0233] Alternatively, the target inference model determines the retrieval modality based on the type of knowledge gap in the target knowledge and generates a corresponding retrieval index. Then, the model retrieves the top N candidate results. For the image information of each candidate reference image, the model calculates its similarity score with the retrieval index and evaluates its information noise level. Finally, the model selects the candidate with moderate similarity and lowest information noise as the image information of the target reference image.

[0234] Another alternative approach is to use the principle of information complementarity for multi-reference fusion. The target reasoning model can select image information from two or more candidate reference images, each reference providing different aspects of the target knowledge gap, and these multi-source reference information are merged in the integration phase.

[0235] Another alternative approach is to employ a reordering strategy. After obtaining candidate references through initial retrieval, the target inference model uses its visual language understanding capabilities to perform in-depth analysis on each candidate, extracting the local regions most relevant to the target knowledge gap, and reordering the candidates based on the alignment of these local regions.

[0236] Step 306: Utilize the target reasoning model to perform a reference integration task. Adjust the initial content based on the target knowledge gap corresponding to the initial content and the image information of the target reference image corresponding to the initial content to obtain the target reasoning content.

[0237] The target inference content is the enhanced output of the target inference model after passing through the search gating task, reference filtering task, and reference integration task. Specifically, the target inference content is supplementary information obtained by semantically describing the image information of the target reference image on top of the initial content. It may include guidance on the use of the reference information (such as explicitly marking the attributes to be borrowed and the parts that can be freely modified). The target inference content can be used to replace the initial content as input to the target multimodal generation model or downstream processing flow. Furthermore, the target inference content can be in the form of natural language text that integrates the original initial content and supplementary knowledge obtained from external retrieval.

[0238] In practical applications, there are multiple ways to perform reference integration tasks and obtain target reasoning content.

[0239] In one alternative approach, the target inference model extracts key visual attributes from the image information of the target reference image using natural language descriptions, and then adds these descriptions as parenthetical phrases or modifiers to the corresponding positions in the initial content to form enhanced inference content.

[0240] Another alternative approach is to attach separate usage guidance paragraphs to the target reasoning content to clearly distinguish between the parts that need to be referenced and the parts that can be generated freely in a structured manner.

[0241] Another alternative approach is that when there are multiple target knowledge gaps or multiple target reference images, the target reasoning model can fuse all the reference information into a coherent narrative, ensuring that the fused description does not produce contradictions or redundancy.

[0242] The embodiments in this specification employ a search gating task to identify candidate knowledge gaps based on initial content and determine the target knowledge gap from among them. This ensures that the retrieval operation is performed only on the filtered target knowledge gap, avoiding concept erosion caused by indiscriminate retrieval. The reference filtering task retrieves candidate reference images based on the target knowledge gap and determines the image information of the target reference image, achieving effective referencing and preventing copying effects caused by reference. The reference integration task adjusts the initial content based on the target knowledge gap and the image information of the target reference image to obtain the target inference content, achieving effective adjustment of the initial content while avoiding concept erosion or copying effects caused by excessive retrieval.

[0243] The following is in conjunction with the appendix Figure 4This document will further illustrate the application of the model training method provided in this manual to content generation scenarios, using it as an example. Figure 4 A flowchart of a multimodal content generation method provided in one embodiment of this specification includes the following steps: Step 402: Obtain the prompt words to be generated, the target inference model, and the target multimodal generation model, wherein the target inference model and the target multimodal generation model are obtained according to the model training method; The prompts to be generated are textual information input into the target inference model during the practical application phase, describing the multimodal content to be generated. Specifically, the prompts to be generated can be short descriptive phrases or complex sentences containing multiple entities, attributes, and relational constraints, and their content may involve general visual concepts or long-tail knowledge entities. The prompts to be generated can be used as input to the target inference model for knowledge gap analysis, retrieval decisions, and prompt integration, driving the entire multimodal content generation process. Furthermore, since the target inference model has already been co-trained, the prompts to be generated can come directly from real user requests without additional preprocessing.

[0244] Step 404: Using the target reasoning model, reason based on the prompt words to be generated to obtain reference prompt words; Reference prompts are enhanced text specifications output by the target inference model after reasoning about the prompts to be generated, supplemented with knowledge gaps and integrated with reference information. Specifically, reference prompts can include the complete or rewritten content of the original prompts to be generated, a semantic description of the identified target knowledge gaps, and reference information extracted from the search results and converted into natural language. Reference prompts can be used as conditional inputs to the target multimodal generative model, replacing the original prompts to be generated. Furthermore, reference prompts can contain explicit or implicit usage guidance information, indicating which attributes the generator must strictly follow the references for and which parts can be freely developed, thus supplementing missing knowledge while preserving space for creative generation.

[0245] Step 406: Using the target multimodal generation model, generate target multimodal content based on the reference prompt words.

[0246] The target multimodal content is the final output generated by the target multimodal generation model based on the reference prompts, and its form corresponds to the type of the target multimodal generation model. Specifically, when the target multimodal generation model is an image generation model, the target multimodal content can be one or more high-resolution images; when the target multimodal generation model is a multimodal generation model, the target multimodal content can also include image sequences, mixed image and text content, or other forms of media data. The target multimodal content can be used as the final generated result returned to the user, satisfying the user's generation needs for specific visual concepts or scenes. The quality of the target multimodal content should be superior to the content generated directly using the original prompts.

[0247] This specification's embodiments utilize a target inference model to infer from the proposed prompts, transforming them into knowledge-enhanced reference prompts. Specifically, the target inference model and target multimodal generation model, obtained through model training methods, calibrate the search strategy of the target inference model with the capabilities of the target multimodal generation model. When faced with proposed prompts, the target inference model can accurately identify prompts exceeding the generation capabilities of the target multimodal generation model, triggering a search and inferring from the proposed prompts to obtain reference prompts. When generating target multimodal content based on these reference prompts, the target multimodal generation model accurately understands the semantic descriptions within the reference prompts and correctly applies usage guidance, thereby generating high-quality target multimodal content and significantly improving the quality of content generated for user requests in real-world application scenarios.

[0248] One optional embodiment of this specification utilizes a target reasoning model to perform reasoning based on the prompt words to be generated, and obtains reference prompt words, including: The target reasoning model is used to perform the search decision gating task. Based on the prompt words to be generated, multiple candidate knowledge gaps corresponding to the prompt words to be generated are obtained. The target knowledge gap corresponding to the prompt words to be generated is determined from the candidate knowledge gaps. The target reasoning model is used to perform a reference filtering task. Based on the target knowledge gap corresponding to the prompt word to be generated, the image information of multiple candidate reference images corresponding to the prompt word to be generated is retrieved, and the image information of the target reference image corresponding to the prompt word to be generated is determined from the image information of the candidate reference images corresponding to the prompt words to be generated. The target reasoning model is used to perform a reference integration task. Based on the target knowledge gap corresponding to the prompt word to be generated and the image information of the target reference image corresponding to the prompt word to be generated, the prompt word to be generated is adjusted to obtain the reference prompt word.

[0249] The candidate knowledge gaps corresponding to the prompt words to be generated are a set of potential concepts identified by the target inference model after semantic analysis of the prompt words, which may exceed the parameterized knowledge range of the target multimodal generative model. Specifically, the candidate knowledge gaps corresponding to the prompt words to be generated may include specific entities involved in the prompt words, the attributes of those entities, spatial or relational constraints between entities, etc., and each candidate knowledge gap is associated with an influence evaluation value. The candidate knowledge gaps corresponding to the prompt words to be generated can be used as the screening objects for the search decision gating task, and only a portion of these gaps will be identified as the target knowledge gaps corresponding to the prompt words to be generated.

[0250] The target knowledge gap corresponding to the prompt term is a key knowledge concept selected from the candidate knowledge gaps corresponding to the prompt term, deemed worthy of triggering the actual retrieval operation. Specifically, the target knowledge gap corresponding to the prompt term is a visual concept whose influence exceeds a preset threshold and which the target multimodal generation model cannot accurately generate or may generate incorrectly, such as the appearance details of a specific character or the precise style of a cultural symbol. The target knowledge gap corresponding to the prompt term can be used as the basis for generating retrieval queries in the reference filtering task, driving the external retrieval system to return relevant reference content. Since the target inference model has been co-trained with the target multimodal generation model, the target knowledge gaps it selects are highly matched with the actual knowledge boundary of the generator.

[0251] The image information of the candidate reference images corresponding to the prompt words to be generated is multiple reference contents returned by an external retrieval interface that are related to the target knowledge gap corresponding to the prompt words to be generated. Specifically, the image information of the candidate reference images corresponding to the prompt words to be generated may include image pixel data, image thumbnails, source URLs, or relevant descriptive information extracted from web page text, which may contain effective signals directly related to the target knowledge gap and information noise unrelated to the gap. The image information of the candidate reference images corresponding to the prompt words to be generated can be used as input for the reference filtering task, from which the target inference model needs to select the most suitable reference content. The quality and relevance of these candidate reference information directly affect the effect of the subsequent reference integration task.

[0252] The image information of the target reference image corresponding to the prompt word to be generated is selected from the image information of the candidate reference images corresponding to the prompt word to be generated, based on the principle of information complementarity, and is used to fill the target knowledge gap corresponding to the prompt word to be generated. Specifically, the image information of the target reference image corresponding to the prompt word to be generated should only include visual attributes directly related to the target knowledge gap, while discarding background, pose, or other object information unrelated to the gap. The image information of the target reference image corresponding to the prompt word to be generated can be used as an information source for the reference integration task and incorporated into the prompt word to be generated.

[0253] This embodiment transforms the prompts to be generated in the practical application stage into knowledge-enhanced reference prompts by sequentially executing a search decision gating task, a reference filtering task, and a reference integration task. Specifically, the gating task identifies candidate knowledge gaps and determines the target knowledge gap based on the prompts to be generated, ensuring that the retrieval operation is triggered only on gaps with high influence outside the generator's knowledge boundary, avoiding meaningless retrieval on concepts already internalized by the generator, thus preventing concept erosion. The filtering task retrieves image information of candidate reference images based on the target knowledge gap and determines the image information of the target reference image. It selects references that can fill specific gaps while minimizing irrelevant content based on the principle of information complementarity, avoiding the copy effect caused by references with excessively high overall matching. The integration task adjusts the prompts to be generated based on the target knowledge gap and the image information of the target reference image to obtain reference prompts. These reference prompts contain guidance on the use of reference information and forcibly discard pixel-level noise by using natural language descriptions as an information bottleneck. Because the target inference model has been co-trained with the target multimodal generation model, the gating, filtering, and integration decisions of the model throughout the inference process are highly matched with the actual knowledge boundary of the current generator. The final reference prompts can inject missing long-tail visual knowledge into the generation conditions in a semantic form without introducing conceptual erosion and copying effects, providing precise guidance for the high-quality output of the subsequent target multimodal generation model.

[0254] Figure 5 A flowchart of an inference model training method provided in one embodiment of this specification includes the following steps: Step 502: Obtain sample prompts, target image generation model, and pre-trained inference model.

[0255] Sample prompts are textual information used during the training phase to describe the content of the image to be generated. They have the same meaning as the sample prompts in claim 1 and are used here to trigger the pre-trained inference model to generate sample inference trajectories. Specifically, a sample prompt can be a phrase describing a specific scene, object, or style, or a complete sentence containing multiple entities and their attribute relationships. Its content can cover general objects or long-tailed visual concepts. Sample prompts can be used as input to the pre-trained inference model to drive the model to perform inference tasks such as knowledge gap identification, retrieval decision-making, and reference integration. Furthermore, the sample prompts can come from a pre-constructed training dataset, which can have long-tailed distribution characteristics to simulate visual knowledge bottlenecks in real user requests.

[0256] A target image generation model is a deep generative model trained to generate corresponding images based on text prompts, outputting multimodal content in image form. Specifically, the target image generation model can be a diffusion model, a generative adversarial network, or a vector quantization-based generative model, and its parameterized knowledge boundary is determined by the time cutoff and distribution range of its training data. The target image generation model can be used to receive reference prompts and generate corresponding candidate images, which should reflect the visual semantics of the prompts as accurately as possible. Furthermore, the target image generation model can be co-trained with a target inference model to expand its knowledge boundary during training.

[0257] A pre-trained inference model is a visual language model that has been pre-trained under supervised learning and possesses the ability to perform multi-step inference analysis on text prompts, having the same meaning as the pre-trained inference model in claim 1. Specifically, the pre-trained inference model may include a visual language model capable of simultaneously understanding textual semantics and visual concepts, and performing inference tasks such as knowledge gap identification, retrieval query generation, and reference information filtering and integration. The pre-trained inference model can be used to perform multiple inferences on sample prompts, outputting structured sample inference trajectories. Furthermore, the parameters of the pre-trained inference model are updated during supervised fine-tuning training, ultimately obtaining a target inference model that matches the knowledge boundary of the target image generation model.

[0258] Step 504: Using the pre-trained inference model, perform multiple inferences based on the sample prompts to obtain multiple sample inference trajectories.

[0259] A sample inference trajectory is a structured data sequence generated by a pre-trained inference model during a single complete inference process for a sample prompt word, and has the same meaning as the sample inference trajectory in claim 1. Specifically, a sample inference trajectory may include identified knowledge gaps, initiated retrieval queries, reference image information selected from the retrieval results, and target prompt words obtained after integrating and adjusting the original prompt words. The sample inference trajectory can be used as conditional input for a target image generation model to generate sample candidate images. Furthermore, different sample inference trajectories correspond to different search decision combinations, such as whether to trigger a search, which references to select, and how to describe the reference information.

[0260] Step 506: Using the target image generation model, generate candidate images of samples corresponding to the multiple sample inference trajectories based on the multiple sample inference trajectories.

[0261] The sample candidate image is an image generated by the target image generation model based on a specific sample inference trajectory, and has the same meaning as the sample candidate image in claim 1. Specifically, the sample candidate image can be a pixel matrix with a preset resolution, and its content should correspond to the adjusted prompt word in the sample inference trajectory. The sample candidate image can be used as an object for subsequent advantage evaluation to quantify the contribution of the inference trajectory to the generation quality. Furthermore, for multiple sample inference trajectories of the same sample prompt word, the target image generation model can generate multiple corresponding sample candidate images respectively.

[0262] Step 508: Perform advantage evaluation on each candidate image to obtain the relative advantage value of each candidate image.

[0263] Advantage assessment is a process of quantitatively scoring the quality of candidate images. This process is based on pre-defined assessment dimensions and an assessment model, and has the same meaning as the advantage assessment in claim 1. Specifically, advantage assessment can employ a pre-trained advantage assessment model, which can be a visual language model capable of scoring images in multiple dimensions based on sample prompts and reference information in sample inference trajectories. Advantage assessment can be used to generate a relative advantage value for each candidate image, thereby distinguishing the superiority or inferiority of different inference trajectories.

[0264] The relative advantage value is a numerical value that measures the degree of quality difference between a candidate image and other candidate images in the same group, and has the same meaning as the relative advantage value in claim 1. Specifically, the relative advantage value can be a value obtained by calculating the difference between the score of an image and the average score within the group among multiple candidate images generated from the same sample prompt word, or by sorting and normalizing. The relative advantage value can be used to indicate whether the corresponding sample inference trajectory has a positive or negative impact on the generation quality. Further, a positive relative advantage value indicates that the corresponding sample inference trajectory is better than the average level of the group, while a negative value indicates that it is worse than the average level.

[0265] Step 510: Based on the relative advantage value, select the positive advantage sample inference trajectory from the multiple sample inference trajectories.

[0266] Positive dominance sample inference trajectories are those sample inference trajectories whose corresponding candidate images have a positive relative dominance value, and have the same meaning as the positive dominance sample inference trajectory in claim 1. Specifically, a positive dominance sample inference trajectory can be the portion of trajectories selected from multiple sample inference trajectories that can guide the target image generation model to produce higher quality images. Positive dominance sample inference trajectories can be used as positive samples for supervised fine-tuning training to update the parameters of the pre-trained inference model. Furthermore, positive dominance sample inference trajectories include correct search decisions (e.g., accurately identifying the knowledge gap to be retrieved, selecting complementary references, and reasonably integrating reference information) and error avoidance (e.g., avoiding unnecessary searches and avoiding excessive copying).

[0267] Step 512: Based on the positive dominance sample inference trajectory, perform supervised fine-tuning training on the pre-trained inference model to obtain the target inference model.

[0268] Supervised fine-tuning training is the process of updating the parameters of a pre-trained model using labeled positive example data, and has the same meaning as the supervised fine-tuning training in claim 1. Specifically, supervised fine-tuning training can employ the cross-entropy loss function, using the decision sequence in the positive-dominance sample inference trajectory as a supervision signal to optimize the output distribution of the pre-trained inference model. Supervised fine-tuning training can be used to enable the inference model to learn which search decisions can produce high-quality generated results, thereby improving its inference ability on unseen prompt words. Furthermore, supervised fine-tuning training uses only positive-dominance sample inference trajectories, discarding negative-dominance trajectories, thereby reinforcing correct decision patterns.

[0269] The target inference model is an inference model obtained through supervised fine-tuning training, capable of adaptively making search decisions based on input prompts and the current image generator capabilities, and has the same meaning as the target inference model in claim 1. Specifically, the target inference model can be used in the practical application stage to perform gating judgment, reference filtering, and integration on the prompts to be generated, and output adjusted reference prompts. The knowledge boundary of the target inference model matches the knowledge boundary of the target image generation model used during training, and can naturally learn from the reward signal when to trigger a search and when to abandon the search without explicit boundary labels.

[0270] This specification describes an embodiment that acquires sample prompts, a target image generation model, and a pre-trained inference model. The pre-trained inference model performs multiple inferences based on the sample prompts to obtain multiple sample inference trajectories, automatically generating diverse sample inference trajectories without manual annotation. The target image generation model generates corresponding candidate images based on each sample inference trajectory, quantifying the actual generation quality of each trajectory through image representation. By evaluating the relative advantage of each candidate image, the relative advantage value is obtained, allowing inference trajectories from different prompts to be compared on a uniform relative scale, accurately distinguishing which inference trajectories produce superior generation results. Based on this relative advantage value, positively dominant sample inference trajectories are selected from the multiple inference trajectories, retaining those that can guide the generation of higher-quality images. The pre-trained inference model is then supervised and fine-tuned based on the positively dominant sample inference trajectories, enabling it to naturally learn the target image generation model's capabilities during training without explicit annotation, and providing reasonable inferences about the sample prompts. This achieves calibration between the target inference model's search strategy and the target image generation model's capabilities.

[0271] Figure 6 A flowchart of an image generation method provided in one embodiment of this specification specifically includes the following steps: Step 602: Obtain the prompt word to be generated, the target inference model, and the target image generation model, wherein the target inference model and the target image generation model are obtained according to the inference model training method.

[0272] The prompts to be generated are textual information describing the content of the desired generated image, input into the target inference model during the actual image generation application stage. Specifically, the prompts can be short descriptive phrases or complex sentences containing multiple entities, attributes, and relational constraints, potentially involving general visual concepts or long-tail knowledge entities. The prompts can be used as input to the target inference model for knowledge gap analysis, retrieval decisions, and prompt integration, driving the entire image generation process. Furthermore, the prompts can come directly from real user requests without additional preprocessing, as the target inference model has already been co-trained with the target image generation model.

[0273] The target inference model is an inference model trained using inference model training methods. It is capable of adaptive search decisions based on input prompts and outputting enhanced reference prompts. Specifically, the target inference model can be a visual language model that has undergone supervised pre-training and rejection sampling fine-tuning. Internally, it has learned gating, filtering, and integration strategies that match the current knowledge boundaries of the target image generation model. The target inference model can be used in the actual image generation stage to receive prompts to be generated, sequentially execute search decision gating tasks, reference filtering tasks, and reference integration tasks, and output reference prompts supplemented with external knowledge. Furthermore, the target inference model can automatically determine when to trigger retrieval and when to abandon retrieval, and integrate the retrieved reference information into the prompts using natural language descriptions, thereby avoiding concept erosion and copying effects.

[0274] The target image generation model is a generative model obtained through inference model training methods, capable of generating corresponding images based on text prompts. Specifically, the target image generation model can be a diffusion model or a generative adversarial network trained with preference optimization. Its parameterized knowledge boundary is expanded compared to the initial model, and it possesses a certain internalization and generation capability for long-tail knowledge supplemented through retrieval during training. The target image generation model can be used to receive reference prompts output by the target inference model in the actual image generation stage and generate the final target image based on these reference prompts.

[0275] Step 604: Using the target reasoning model, reason based on the prompt words to be generated to obtain reference prompt words.

[0276] Reference prompts are enhanced text specifications output by the target inference model after reasoning about the prompts to be generated, supplemented with knowledge gaps and integrated with reference information. Specifically, reference prompts can include the complete or rewritten content of the original prompts to be generated, a semantic description of the identified target knowledge gaps, and reference information extracted from the retrieval results and converted into natural language. Reference prompts can be used as conditional input to the target image generation model, replacing the original prompts to be generated. Furthermore, reference prompts can contain explicit or implicit usage guidance information, instructing the image generator which attributes must strictly follow the reference and which parts can be freely developed, thus supplementing missing knowledge while preserving space for creative generation.

[0277] Step 606: Using the target image generation model, generate the target image based on the reference prompt words.

[0278] The target image is the output image generated by the target image generation model based on the reference prompts, and it is in the form of a pixel matrix. Specifically, the target image can be a high-resolution color image, and its content should accurately reflect the semantic constraints in the reference prompts, including entities, attributes, relationships in the original prompts to be generated, as well as visual details supplemented from external retrieval. The target image can be used as the final generated result returned to the user, satisfying the user's needs for generating specific visual concepts or scenes.

[0279] This embodiment of the specification, by acquiring the prompt words to be generated, the target inference model, and the target image generation model, can efficiently process user image generation requests containing long-tail visual knowledge in practical application scenarios by leveraging the collaborative calibration relationship established during the training phase. The target inference model has already established an awareness of the generation capabilities of the target image generation model, and the reference prompt words obtained thereby already contain accurate semantic descriptions and usage guidance for the missing knowledge. Subsequently, the target image generation model generates the target image based on the reference prompt words. Since the target image generation model has achieved joint calibration with the target inference model during the training phase, the generator can accurately understand the semantic descriptions in the reference prompt words and correctly apply the borrowing guidance, ultimately generating a high-quality target image that contains accurate long-tail visual details while retaining creative composition.

[0280] Figure 7 A flowchart illustrating the processing steps of an image generation method provided in one embodiment of this specification includes: Based on the flowchart shown in the accompanying drawings, embodiments of this specification provide an image generation method driven by knowledge gaps. Specifically, firstly, user-inputted prompt words are obtained. A visual language model inferencer performs semantic analysis on these prompt words to identify existing knowledge gaps. The identified knowledge gaps are further classified, including determining their type and assessing their severity. For each knowledge gap, the inferencer generates a search query with a modality label indicating whether an image search or a text search modality should be used.

[0281] Then, a gating decision is executed: if the content corresponding to the knowledge gap is critical / important to the image generator, and the gating output is negative, the search operation is not triggered, and the process skips the retrieval step directly; if the gating output is positive, the search operation is triggered. After the search is triggered, the search results are obtained from an external retrieval system based on the search query. These results contain multiple candidate reference images or text descriptions.

[0282] Next, a filtering operation is performed: target references that can fill the specific knowledge gap and contain the least amount of irrelevant content are selected from the search results. The filtering principle is information complementarity, that is, only target references directly related to the gap are retained, while references with too high an overall match that may cause a copying effect, as well as references that carry background noise unrelated to the gap, are discarded.

[0283] The filtered target references enter the integration phase. The integration process merges the original prompts to be generated, the classification information of knowledge gaps, and the target references into an enhanced text specification. This specification routes visual reference information through natural language descriptions, while explicitly annotating the attributes to be borrowed from the references and the parts that can be freely generated. The enhanced text specification can take the form of "Basic description: ...; Borrowed references: ...; Free to develop: ...".

[0284] Finally, the enhanced text specification is input into the image generator, which generates the final image based on the semantic description and usage guidelines in the specification. The generated image can be evaluated by an automatic scoring system, scoring it based on multiple dimensions such as checklist compliance, rubric scoring, and cue word follow-through, to verify the generation quality. Through the above process, retrieval is triggered only for knowledge gaps outside the generator's knowledge boundaries. By using natural language routing for visual references and explicit attribute annotation, conceptual erosion and copying effects are effectively avoided, improving the accuracy of long-tail visual knowledge generation.

[0285] Corresponding to the above method embodiments, this specification also provides embodiments of a model training device. Figure 8 This is a schematic diagram of a model training device provided in one embodiment of this specification. Figure 8 As shown, the device includes: The first acquisition module 802 is configured to acquire sample prompt words, target multimodal generation model and pre-trained inference model; The first reasoning module 804 is configured to use a pre-trained reasoning model to perform multiple inferences based on sample prompts and obtain multiple sample reasoning trajectories. The first candidate image generation module 806 is configured to use the target multimodal generation model to generate sample candidate images corresponding to multiple sample inference trajectories based on multiple sample inference trajectories; The first advantage evaluation module 808 is configured to evaluate the advantage of each candidate image and obtain the relative advantage value of each candidate image. The first screening module 810 is configured to select positive advantage sample inference trajectories from multiple sample inference trajectories based on the relative advantage value. The first supervised fine-tuning module 812 is configured to perform supervised fine-tuning training on the pre-trained inference model based on the positive dominance sample inference trajectory to obtain the target inference model.

[0286] Optionally, the model training device further includes an inference model pre-training module, configured to acquire pre-training prompts, label inference trajectories corresponding to the pre-training prompts, and an initial inference model; use the initial inference model to perform at least one task inference based on the pre-training prompts to obtain an initial inference trajectory; and perform supervised pre-training on the initial inference model based on the initial inference trajectory and the label inference trajectory to obtain a pre-trained inference model.

[0287] Optionally, the inference model pre-training module is further configured to perform a search decision gating task using the pre-trained inference model, obtain multiple candidate knowledge gaps based on pre-trained prompts, and determine the target knowledge gap from the candidate knowledge gaps; perform a reference filtering task using the pre-trained inference model, retrieve image information of multiple candidate reference images based on the target knowledge gap, and determine the image information of the target reference image from the image information of the multiple candidate reference images; perform a reference integration task using the pre-trained inference model, adjust the pre-trained prompts based on the target knowledge gap and the image information of the target reference image to obtain the target prompts; and determine the initial inference trajectory based on the target knowledge gap, the image information of the target reference image, and the target prompts.

[0288] Optionally, the inference model pre-training module is further configured to use the pre-trained inference model to generate a retrieval index of the target modality corresponding to the target knowledge gap, based on the knowledge type of the target knowledge gap; Optionally, the inference model pre-training module is further configured to retrieve image information of multiple candidate reference images of the target modality based on the retrieval index of the target modality.

[0289] Optionally, the inference model pre-training module is further configured to determine the image information of the target reference image from the image information of the multiple candidate reference images based on the similarity between the retrieval index of the target modality and the image information of the multiple candidate reference images, as well as the information noise of the multiple candidate reference images.

[0290] Optionally, the inference model pre-training module is further configured to use the pre-trained inference model to perform semantic analysis on the pre-trained prompt words to obtain semantic analysis results; based on the semantic analysis results, determine at least one candidate knowledge gap; and based on the influence degree of the at least one candidate knowledge gap, determine the target knowledge gap, wherein the influence degree is the degree of influence of the knowledge gap on image generation.

[0291] Optionally, the inference model pre-training module is further configured to perform a reference integration task using the pre-trained inference model, adjusting the pre-trained prompts based on semantic analysis results, target knowledge gaps, and image information of the target reference image to obtain target prompts, wherein the target prompts include guidance information on the use of image information of the target reference image.

[0292] Optionally, the inference model pre-training module is further configured to perform a search decision gating task using the pre-trained inference model, obtain at least one candidate knowledge gap based on the pre-trained prompts, and discard at least one candidate knowledge gap if the influence of at least one candidate knowledge gap is less than a preset influence threshold; and generate an initial inference trajectory based on the execution results of the pre-trained prompts and the search decision gating task.

[0293] Optionally, the model training device further includes a multimodal generative model pre-training module, configured to acquire sample generation prompts, an initial multimodal generative model, and a pre-trained inference model; use the pre-trained inference model to perform inference based on the sample generation prompts to obtain a pre-trained inference trajectory; use the initial multimodal generative model to sample and generate multiple sampled candidate images based on the pre-trained inference trajectory; perform evaluation protocol scoring on each sampled candidate image to obtain an evaluation score for each sampled candidate image; determine image preference pairs from each sampled candidate image based on the evaluation scores; and perform preference optimization training on the initial multimodal generative model based on the image preference pairs until the target multimodal generative model is obtained.

[0294] Optionally, the multimodal generative model pre-training module is further configured to determine positive and negative preference images from each sampled candidate image based on the evaluation score; Optionally, the multimodal generative model pre-training module is further configured to construct a preference contrast loss based on positive preference images and negative preference images; and to perform preference optimization training on the initial multimodal generative model based on the preference contrast loss until the target multimodal generative model is obtained.

[0295] Optionally, the first advantage evaluation module 808 is further configured to perform advantage evaluation on each sample candidate image based on a pre-trained advantage evaluation model to obtain the relative advantage value of each sample candidate image. The pre-trained evaluation dimensions include at least one of checklist compliance, metric score, cue word follow-through, text rendering effect, image artifacts, visual reference fidelity, and text reference fidelity.

[0296] Optionally, the first acquisition module 802 is further configured to acquire initial prompt words; perform semantic analysis on the initial prompt words to obtain prompt word entities; determine the label images corresponding to each prompt word entity based on each prompt word entity; and generate sample prompt words using a preset prompt word generation model based on the label images corresponding to each prompt word entity and each prompt word entity.

[0297] Optionally, the first acquisition module 802 is further configured to annotate the prompt word entities to obtain annotated prompt words; perform statistics on the annotated prompt word entities to obtain the proportion of each entity in the sample set; construct the estimated training set frequency of the long-tail distribution based on the proportion of each entity in the sample set; and filter each entity through the estimated training set frequency of the long-tail distribution to obtain the filtered prompt word entities.

[0298] This specification describes an embodiment that acquires sample prompts, a target multimodal generative model, and a pre-trained inference model. The pre-trained inference model performs multiple inferences based on the sample prompts to obtain multiple sample inference trajectories, allowing it to explore different search decisions based on its own knowledge boundaries. The target multimodal generative model generates corresponding sample candidate images based on each sample inference trajectory; the output quality directly reflects the fit between the search behavior decided by the inference trajectory and the knowledge boundary of the target multimodal generative model. Each sample candidate image is evaluated for its relative advantage value. Based on this relative advantage value, positive advantage sample inference trajectories are selected from the multiple sample inference trajectories, retaining those that improve the quality of the generated images and discarding those that degrade the quality. The pre-trained inference model is then subjected to supervised fine-tuning training based on the positive advantage sample inference trajectories to obtain the target inference model. The inference model can naturally learn the knowledge boundary of the enhanced target multimodal generative model from the reward signal without any explicit knowledge boundary labels, achieving joint calibration of the target inference model's search strategy and the generator's capabilities.

[0299] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0300] Corresponding to the above method embodiments, this specification also provides embodiments of content reasoning apparatus. Figure 9 This is a schematic diagram of a content reasoning device provided in one embodiment of this specification. Figure 9 As shown, the device includes: The gating task module 902 is configured to input initial content into the target inference model, perform a search gating task, obtain multiple candidate knowledge gaps corresponding to the initial content, and determine the target knowledge gap corresponding to the initial content from the candidate knowledge gaps. The reference filtering module 904 is configured to perform a reference filtering task using a target inference model, retrieve image information of multiple candidate reference images corresponding to the initial content based on the target knowledge gap corresponding to the initial content, and determine the image information of the target reference image corresponding to the initial content from the image information of the multiple candidate reference images corresponding to the initial content. The reference integration module 906 is configured to perform a reference integration task using the target inference model, adjusting the initial content based on the target knowledge gap corresponding to the initial content and the image information of the target reference image corresponding to the initial content, to obtain the target inference content.

[0301] The embodiments in this specification employ a search gating task to identify candidate knowledge gaps based on initial content and determine the target knowledge gap from among them. This ensures that the retrieval operation is performed only on the filtered target knowledge gap, avoiding concept erosion caused by indiscriminate retrieval. The reference filtering task retrieves candidate reference images based on the target knowledge gap and determines the image information of the target reference image, achieving effective referencing and preventing copying effects caused by reference. The reference integration task adjusts the initial content based on the target knowledge gap and the image information of the target reference image to obtain the target inference content, achieving effective adjustment of the initial content while avoiding concept erosion or copying effects caused by excessive retrieval.

[0302] Corresponding to the above method embodiments, this specification also provides embodiments of a multimodal content generation apparatus. Figure 10 This is a schematic diagram of a multimodal content generation apparatus provided in one embodiment of this specification. Figure 10 As shown, the device includes: The second acquisition module 1002 is configured to acquire the prompt word to be generated, the target reasoning model, and the target multimodal generation model, wherein the target reasoning model and the target multimodal generation model are obtained according to the model training method; The second reasoning module 1004 is configured to use the target reasoning model to reason based on the prompt words to be generated, and obtain reference prompt words. The multimodal generation module 1006 is configured to generate target multimodal content based on reference prompts using the target multimodal generation model.

[0303] Optionally, the second reasoning module 1004 is further configured to perform a search decision gating task using a target reasoning model, obtain multiple candidate knowledge gaps corresponding to the prompt words to be generated based on the prompt words to be generated, and determine the target knowledge gap corresponding to the prompt words to be generated from the candidate knowledge gaps; perform a reference filtering task using the target reasoning model, retrieve image information of multiple candidate reference images corresponding to the prompt words to be generated based on the target knowledge gap corresponding to the prompt words to be generated, and determine the image information of the target reference image corresponding to the prompt words to be generated from the image information of the candidate reference images corresponding to the prompt words to be generated; and perform a reference integration task using the target reasoning model, adjust the prompt words to be generated based on the target knowledge gap corresponding to the prompt words to be generated and the image information of the target reference image corresponding to the prompt words to be generated, to obtain reference prompt words.

[0304] This embodiment transforms the prompts to be generated in the practical application stage into knowledge-enhanced reference prompts by sequentially executing a search decision gating task, a reference filtering task, and a reference integration task. Specifically, the gating task identifies candidate knowledge gaps and determines the target knowledge gap based on the prompts to be generated, ensuring that the retrieval operation is triggered only on gaps with high influence outside the generator's knowledge boundary, avoiding meaningless retrieval on concepts already internalized by the generator, thus preventing concept erosion. The filtering task retrieves image information of candidate reference images based on the target knowledge gap and determines the image information of the target reference image. It selects references that can fill specific gaps while minimizing irrelevant content based on the principle of information complementarity, avoiding the copy effect caused by references with excessively high overall matching. The integration task adjusts the prompts to be generated based on the target knowledge gap and the image information of the target reference image to obtain reference prompts. These reference prompts contain guidance on the use of reference information and forcibly discard pixel-level noise by using natural language descriptions as an information bottleneck. Because the target inference model has been co-trained with the target multimodal generation model, the gating, filtering, and integration decisions of the model throughout the inference process are highly matched with the actual knowledge boundary of the current generator. The final reference prompts can inject missing long-tail visual knowledge into the generation conditions in a semantic form without introducing conceptual erosion and copying effects, providing precise guidance for the high-quality output of the subsequent target multimodal generation model.

[0305] The above is a schematic scheme of a modal content generation device according to this embodiment. It should be noted that the technical solution of this modal content generation device and the technical solution of the modal content generation method described above belong to the same concept. For details not described in detail in the technical solution of the modal content generation device, please refer to the description of the technical solution of the modal content generation method described above.

[0306] Corresponding to the above method embodiments, this specification also provides embodiments of inference model training devices. Figure 11 This is a schematic diagram of a reasoning model training device provided in one embodiment of this specification. Figure 11 As shown, the device includes: The third acquisition module 1102 is configured to acquire sample prompt words, target image generation model and pre-trained inference model; The third reasoning module 1104 is configured to use a pre-trained reasoning model to perform multiple inferences based on sample prompts and obtain multiple sample reasoning trajectories. The second candidate image generation module 1106 is configured to use the target image generation model to generate sample candidate images corresponding to multiple sample inference trajectories based on multiple sample inference trajectories. The second advantage evaluation module 1108 is configured to evaluate the advantage of each candidate image and obtain the relative advantage value of each candidate image. The second screening module 1110 is configured to select positive advantage sample inference trajectories from multiple sample inference trajectories based on the relative advantage value. The second supervised fine-tuning module 1112 is configured to perform supervised fine-tuning training on the pre-trained inference model based on the positive dominance sample inference trajectory to obtain the target inference model.

[0307] This specification's embodiments acquire sample prompts, a target image generation model, and a pre-trained inference model. The pre-trained inference model then performs multiple inferences based on the sample prompts to obtain multiple sample inference trajectories. This enables the automatic generation of diverse search decision paths without relying on manual annotation, covering the entire decision space from triggering retrieval to abandoning the retrieval, and from selecting references to integrating them. The target image generation model generates corresponding candidate images based on each sample inference trajectory, quantifying the actual image generation quality of each trajectory. Image quality becomes a direct basis for evaluating the merits of the inference trajectory. By evaluating the advantages of each candidate image and obtaining relative advantage values, the differences in generation difficulty between different sample prompts can be eliminated. This allows inference trajectories from different prompts to be compared on a unified relative scale, accurately identifying which inference decisions have a positive effect within the knowledge boundary of a specific image generation model. Based on the relative advantage values, positively dominant sample inference trajectories are selected, uncovering decision patterns that guide the image generation model to output high-quality images, while discarding undesirable decisions that lead to concept erosion or copying effects. Finally, the pre-trained inference model is trained under supervision based on the inference trajectory of positive dominance samples, so that the inference model can learn the actual capability boundary of the current target image generation model without explicit knowledge boundary labels.

[0308] The above is an illustrative scheme of a reasoning model training device according to this embodiment. It should be noted that the technical solution of this reasoning model training device and the technical solution of the aforementioned reasoning model training method belong to the same concept. For details not described in detail in the technical solution of the reasoning model training device, please refer to the description of the technical solution of the aforementioned reasoning model training method.

[0309] Corresponding to the above method embodiments, this specification also provides embodiments of an image generation apparatus. Figure 12 This is a schematic diagram of an image generation apparatus provided in one embodiment of this specification. Figure 12 As shown, the device includes: The fourth acquisition module 1202 is configured to acquire the prompt word to be generated, the target inference model and the target image generation model, wherein the target inference model and the target image generation model are obtained according to the inference model training method; The fourth reasoning module 1204 is configured to use the target reasoning model to reason based on the prompt words to be generated, and obtain reference prompt words; The image generation module 1206 is configured to generate a target image based on reference prompt words using a target image generation model.

[0310] This specification's embodiments, by acquiring the prompt words to be generated, the target inference model, and the target image generation model, can efficiently process user image generation requests containing long-tail visual knowledge in practical application scenarios by leveraging the collaborative calibration relationship established during the training phase. The target inference model uses the prompt words to generate to obtain reference prompt words. This inference process automatically executes a three-stage noise suppression search protocol: the gating stage triggers retrieval only for gaps that meet the impact threshold to prevent concept erosion; the filtering stage selects references based on the principle of information complementarity to suppress copying effects; and the integration stage routes visual references through natural language to discard pixel-level noise. The resulting reference prompt words already contain accurate semantic descriptions and usage guidance for the missing knowledge, without directly transmitting high-dimensional pixel noise. Subsequently, the target image generation model uses the reference prompt words to generate the target image. Because the target image generation model has expanded its knowledge boundary through preference optimization during the training phase and has achieved joint calibration with the target inference model, the generator can accurately understand the semantic descriptions in the reference prompt words and correctly apply the borrowing guidance, ultimately generating a high-quality target image that contains accurate long-tail visual details while retaining creative composition.

[0311] The above is an illustrative scheme of an image generation apparatus according to this embodiment. It should be noted that the technical solution of this image generation apparatus and the technical solution of the image generation method described above belong to the same concept. For details not described in detail in the technical solution of the image generation apparatus, please refer to the description of the technical solution of the image generation method described above.

[0312] Figure 13 This is a structural block diagram of a computing device according to one embodiment of this specification. The components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0313] The computing device 1300 also includes an access device 1340, which enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1340 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0314] In one embodiment of this specification, the above-described components of the computing device 1300 and Figure 13 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0315] The computing device 1300 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1300 can also be a mobile or stationary server.

[0316] The processor 1320 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of any of the above-mentioned content reasoning method, multimodal content generation method, reasoning model training method, and image generation method.

[0317] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of any of the above-mentioned model training method, content inference method, multimodal content generation method, inference model training method, and image generation method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of any of the above-mentioned model training method, content inference method, multimodal content generation method, inference model training method, and image generation method.

[0318] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described methods: model training method, content reasoning method, multimodal content generation method, reasoning model training method, and image generation method.

[0319] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of any of the above-described model training method, content inference method, multimodal content generation method, inference model training method, and image generation method. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of any of the above-described model training method, content inference method, multimodal content generation method, inference model training method, and image generation method.

[0320] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described model training method, content reasoning method, multimodal content generation method, reasoning model training method, and image generation method.

[0321] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of any of the above-mentioned model training method, content reasoning method, multimodal content generation method, reasoning model training method, and image generation method. Details not described in detail in the technical solution of the computer program product can be found in the descriptions of the technical solutions of any of the above-mentioned model training method, content reasoning method, multimodal content generation method, reasoning model training method, and image generation method.

[0322] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0323] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0324] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0325] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0326] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, comprising: Acquire sample prompts, target multimodal generation model, and pre-trained inference model; Using the pre-trained inference model, multiple inferences are performed based on the sample prompts to obtain multiple sample inference trajectories; Using the target multimodal generation model, candidate image of the sample corresponding to the multiple sample inference trajectories is generated based on the multiple sample inference trajectories; The relative advantage value of each candidate image is obtained by evaluating the advantage of each candidate image. Based on the relative advantage value, positive advantage sample inference trajectories are selected from the multiple sample inference trajectories; Based on the positive dominance sample inference trajectory, the pre-trained inference model is subjected to supervised fine-tuning training to obtain the target inference model.

2. The method according to claim 1, wherein the pre-training step of the pre-trained inference model includes: Obtain pre-trained prompt words, the label inference trajectory corresponding to the pre-trained prompt words, and the initial inference model; Using the initial inference model, at least one task inference is performed based on the pre-trained prompts to obtain an initial inference trajectory; Based on the initial inference trajectory and the labeled inference trajectory, the initial inference model is pre-trained under supervision to obtain a pre-trained inference model.

3. The method according to claim 2, wherein the at least one task includes a search decision gating task, a reference filtering task, and a reference integration task; The step of using the initial inference model to perform at least one task inference based on the pre-trained prompts to obtain an initial inference trajectory includes: The search decision gating task is executed using the pre-trained inference model, and multiple candidate knowledge gaps are obtained based on the pre-trained prompts. The target knowledge gap is then determined from the candidate knowledge gaps. The reference filtering task is performed using the pre-trained inference model. Image information of multiple candidate reference images is retrieved based on the target knowledge gap, and image information of the target reference image is determined from the image information of the multiple candidate reference images. The reference integration task is performed using the pre-trained inference model. The pre-trained prompt words are adjusted based on the target knowledge gap and the image information of the target reference image to obtain the target prompt words. The initial inference trajectory is determined based on the target knowledge gap, the image information of the target reference image, and the target prompt words.

4. The method according to claim 3, wherein the knowledge gap corresponds to a knowledge type, and any knowledge type corresponds to a different modality of retrieval index; Before retrieving image information from multiple candidate reference images based on the target knowledge gap, the method further includes: Using the pre-trained inference model, a retrieval index for the target modality corresponding to the target knowledge gap is generated based on the knowledge type of the target knowledge gap; The step of retrieving image information from multiple candidate reference images based on the target knowledge gap includes: Based on the retrieval index of the target modality, image information of multiple candidate reference images of the target modality is retrieved.

5. The method according to claim 4, wherein the image information carries information noise, and the information noise is information whose matching degree with the retrieval index is lower than a preset threshold; The step of determining the image information of the target reference image from the image information of the plurality of candidate reference images includes: Based on the similarity between the retrieval index of the target modality and the image information of the plurality of candidate reference images, as well as the information noise of the plurality of candidate reference images, the image information of the target reference image is determined from the image information of the plurality of candidate reference images.

6. The method according to claim 3, wherein performing the search decision gating task using the pre-trained inference model, obtaining multiple candidate knowledge gaps based on the pre-trained prompts, and determining the target knowledge gap from the candidate knowledge gaps, comprises: Using the pre-trained inference model, semantic analysis is performed on the pre-trained prompt words to obtain semantic analysis results; Based on the semantic analysis results, at least one candidate knowledge gap is identified; The target knowledge gap is determined based on the influence degree of the at least one candidate knowledge gap, wherein the influence degree is the extent to which the knowledge gap affects image generation.

7. The method according to claim 6, wherein performing the reference integration task using the pre-trained inference model, adjusting the pre-trained prompt words based on the target knowledge gap and the image information of the target reference image to obtain the target prompt words, includes: The reference integration task is performed using the pre-trained inference model. Based on the semantic analysis results, the target knowledge gap, and the image information of the target reference image, the pre-trained prompt words are adjusted to obtain the target prompt words, wherein the target prompt words include guidance information on the use of the image information of the target reference image.

8. The method according to claim 2, wherein the initial inference trajectory includes at least one candidate knowledge gap, and the at least one task includes a search decision gating task; The step of using the initial inference model to perform at least one task inference based on the pre-trained prompts to obtain an initial inference trajectory includes: The search decision gating task is executed using the pre-trained inference model. At least one candidate knowledge gap is obtained based on the pre-trained prompt words. If the influence of at least one candidate knowledge gap is less than a preset influence threshold, the at least one candidate knowledge gap is abandoned. An initial inference trajectory is generated based on the pre-trained prompts and the execution results of the search decision gating task.

9. The method according to claim 1, wherein the training step of the target multimodal generative model includes: Obtain sample-generated prompts, an initial multimodal generation model, and a pre-trained inference model; Using the pre-trained inference model, inference is performed based on the sample to generate prompt words and obtain the pre-trained inference trajectory; Using the initial multimodal generation model, multiple sampled candidate images are generated based on the pre-trained inference trajectory; Each sampled candidate image is evaluated and scored according to the evaluation protocol to obtain an evaluation score for each sampled candidate image; Based on the evaluation scores, image preference pairs are determined from each of the sampled candidate images; Based on the image preference pairs, the initial multimodal generation model is trained with preference optimization until the target multimodal generation model is obtained.

10. The method of claim 9, wherein the image preference pair comprises a positive preference image and a negative preference image; The step of determining image preference pairs from each sampled candidate image based on the evaluation score includes: Based on the evaluation scores, positive preference images and negative preference images are determined from each sampled candidate image; The step of performing preference optimization training on the initial multimodal generation model based on the image preference pairs until the target multimodal generation model is obtained includes: Based on the positive preference image and the negative preference image, a preference contrast loss is constructed; Based on the preference contrast loss, the initial multimodal generation model is trained with preference optimization until the target multimodal generation model is obtained.

11. The method according to claim 1, wherein obtaining sample prompt words includes: Get the initial prompt word; Perform semantic analysis on the initial prompt word to obtain the prompt word entity; Based on each prompt word entity, determine the label image corresponding to each prompt word entity; The prompt word entities are annotated to obtain annotated prompt word entities; The proportion of each labeled prompt word entity in the sample set is obtained by statistically analyzing the labeled prompt word entities. Based on the proportion of each labeled prompt word entity in the sample set, construct the estimated training set frequency of the long-tail distribution; The labeled prompt word entities are filtered by estimating the training set frequency of the long-tail distribution to obtain the filtered prompt word entities; Using a preset prompt word generation model, sample prompt words are generated based on each filtered prompt word entity and the corresponding label image.

12. A content reasoning method, comprising: The initial content is input into the target inference model, and a search gating task is performed to obtain multiple candidate knowledge gaps corresponding to the initial content. The target knowledge gap corresponding to the initial content is determined from the candidate knowledge gaps. The target inference model is obtained by the model training method according to any one of claims 1-11. The target inference model is used to perform a reference filtering task. Based on the target knowledge gap corresponding to the initial content, image information of multiple candidate reference images corresponding to the initial content is retrieved, and image information of the target reference image corresponding to the initial content is determined from the image information of the multiple candidate reference images corresponding to the initial content. The target inference model is used to perform a reference integration task. The initial content is adjusted based on the target knowledge gap corresponding to the initial content and the image information of the target reference image corresponding to the initial content to obtain the target inference content.

13. A method for generating multimodal content, comprising: Obtain the prompt words to be generated, the target inference model, and the target multimodal generation model, wherein the target inference model and the target multimodal generation model are obtained by the model training method according to any one of claims 1-11; Using the target reasoning model, reasoning is performed based on the prompt words to be generated to obtain reference prompt words; Using the target multimodal generation model, target multimodal content is generated based on the reference prompt words.

14. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 13.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 13.

Images