Image generation method and device, electronic equipment and storage medium

Abstract

The invention relates to an image generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring text prompt data and reference image data, inputting the text prompt data and the reference image data into a unified multi-modal understanding and generation model for image generation processing, and outputting to obtain target image data; wherein the unified multi-modal understanding and generation model comprises a display visual planning module and a self-adaptive visual correction module, and the display visual planning module is used for generating a visual inspection instruction based on text prompt data and the reference image data; the visual inspection instruction is used for guiding the self-adaptive visual correction module to carry out self-adaptive correction on the currently generated image data and finally outputting the target image data. By adopting the method, the visual quality and controllability of the image generation content can be remarkably improved.

Description

Image generation methods, apparatus, electronic devices and storage media Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, the Transformer architecture has driven breakthrough successes for large language models (LLMs) in text tasks. Building on this, researchers have extended its capabilities to the multimodal domain, giving rise to unified multimodal understanding and generation models. These models can simultaneously process and generate data such as text and images, providing technical support for complex visual tasks such as text-to-image generation and storyboard creation.

[0003] To improve the quality and controllability of generated content, researchers have introduced thought mechanisms derived from the text domain (such as Chain-of-Thought (CoT)). When this type of mechanism is applied to a unified multimodal understanding and generation model, the model focuses on ensuring semantic consistency of the text, that is, ensuring that the generated content matches the text prompts and descriptions.

[0004] However, this approach fails to explicitly consider and maintain visual context consistency. This results in poor model performance when handling tasks requiring high visual coherence and precise preservation of fine-grained features. It often exhibits significant differences between the generated image and the input visual context, insufficient coherence across frames / images, and the inability to stably maintain features such as character IDs, object textures, and scene layouts. This severely reduces the visual quality of image-generated content for tasks such as storyboard generation and multi-reference image synthesis. Summary of the Invention

[0005] This disclosure provides an image generation method, apparatus, and electronic device to at least address the problem of poor visual quality in image generation content when the unified multimodal understanding and generation model in related technologies handles tasks requiring high visual coherence and accurate preservation of fine-grained features. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, an image generation method is provided, comprising:

[0007] Obtain text prompt data and reference image data;

[0008] The text prompt data and the reference image data are input into the unified multimodal understanding and generation model for image generation processing, and the target image data is output.

[0009] The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module is used to generate visual inspection instructions based on the text prompt data and the reference image data. The visual inspection instructions are used to guide the adaptive visual correction module to adaptively correct the currently generated image data and finally output the target image data.

[0010] In one embodiment, the unified multimodal understanding and generation model further includes an initial generation module. The step of inputting the text prompt data and the reference image data into the unified multimodal understanding and generation module for image generation processing, and outputting target image data, includes:

[0011] The text prompt data and the reference image data are input into the display visual planning module for planning processing, and a visual inspection instruction is output.

[0012] The text prompt data and the reference image data are input into the initial generation module for image generation processing, and the initial draft image data is output.

[0013] The initial draft image data and the visual inspection instructions are input into the adaptive visual correction module for image correction processing, and the target image data is output.

[0014] In one embodiment, the visual inspection instruction includes at least one inspection item, which includes visual elements to be aligned, visual type, and target region. The process of image correction processing performed by the adaptive visual correction module includes:

[0015] For each inspection item in the visual inspection instruction, check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item.

[0016] When inconsistencies are found in the inspection results, an editing instruction is generated to indicate the content that needs to be corrected.

[0017] The initial draft image data is corrected based on the editing instructions, and the corrected image data is used as the new initial draft image data. The process then jumps to each inspection item in the visual inspection instructions to check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item, until the iterative correction conditions are met. Finally, the current initial draft image data is output as the target image data.

[0018] In one embodiment, the training process of the unified multimodal understanding and generative model includes:

[0019] A training set is constructed, and an initial unified multimodal understanding and generation model is trained based on the sample data in the training set to obtain the first unified multimodal understanding and generation model;

[0020] The first unified multimodal understanding and generation model is subjected to reinforcement learning based on the sample data in the training set to obtain the unified multimodal understanding and generation model.

[0021] In one embodiment, the training set includes a first training set and a second training set. The sample data in the first training set includes sample text prompt data and sample reference image data, and the annotation information of the sample data includes ground truth visual inspection instructions and ground truth image data.

[0022] The sample data in the second training set includes sample text prompt data, sample reference image data, defect image data generated based on the sample text prompt data and the sample reference image data, and truth visual inspection instructions. The annotation information of the sample data includes truth editing instructions and truth image data.

[0023] In one embodiment, training the initial unified multimodal understanding and generation model based on sample data in the training set includes:

[0024] Randomly sample data from the first training set and the second training set;

[0025] The sampled data is input into the initial unified multimodal understanding and generation model for image generation processing, and the output is the predicted information;

[0026] The prediction loss is determined based on the prediction information and the annotation information of the sample data, and the parameters of the initial unified multimodal understanding and generation model are tuned based on the prediction loss.

[0027] In one embodiment, when the sample data comes from the first training set, the prediction information includes predicted visual inspection instructions and predicted image data, and determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0028] The first loss is determined based on the predicted visual inspection instruction and the true visual inspection instruction corresponding to the sample data.

[0029] The second loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0030] The predicted loss is determined based on the first loss and the second loss.

[0031] In one embodiment, when the sample data comes from the second training set, the prediction information includes predicted editing instructions and predicted image data, and determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0032] The third loss is determined based on the prediction editing instructions and the truth value editing instructions corresponding to the sample data;

[0033] The fourth loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0034] The predicted loss is determined based on the third loss and the fourth loss.

[0035] In one embodiment, the reinforcement learning of the first unified multimodal understanding and generation model based on sample data in the training set includes:

[0036] The sample text prompt data and sample reference image data in the sample data are input into the first unified multimodal understanding and generation model for image generation processing, and the output is the predicted visual inspection instruction and the predicted image data.

[0037] The target reward value is determined based on the predicted visual inspection instructions, the predicted image data, the sample text prompt data, and the sample reference image data;

[0038] The target reward value is used as the reward signal for reinforcement learning, and the first unified multimodal understanding and generation model is optimized based on the reward signal.

[0039] In one embodiment, determining the target reward value based on the predicted visual inspection instruction and the predicted image data, as well as the sample text prompt data and the sample reference image data, includes:

[0040] A first reward value is determined based on the predicted image data and the sample text prompt data, wherein the first reward value is used to characterize the textual semantic consistency between the predicted image data and the sample text prompt data;

[0041] A second reward value is determined based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data. The second reward value is used to characterize the visual feature consistency between the predicted image data and the sample reference image data.

[0042] The target reward value is determined based on the first reward value and the second reward value.

[0043] In one embodiment, determining the second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data includes:

[0044] For any inspection item in the predicted visual inspection instruction, a reward value measurement model corresponding to the inspection item is determined according to the visual type indicated by the inspection item, and the visual feature consistency between the predicted image data and the sample reference image data is determined using the reward value measurement model to obtain the reward value corresponding to the inspection item.

[0045] The second reward value is determined based on the reward value corresponding to each of the aforementioned inspection items.

[0046] According to a second aspect of the present disclosure, an image generation apparatus is provided, comprising:

[0047] The acquisition unit is configured to acquire text prompt data and reference image data;

[0048] The image generation processing unit is configured to perform image generation processing by inputting the text prompt data and the reference image data into a unified multimodal understanding and generation model, and output the target image data.

[0049] The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module is used to generate visual inspection instructions based on the text prompt data and the reference image data. The visual inspection instructions are used to guide the adaptive visual correction module to adaptively correct the currently generated image data and finally output the target image data.

[0050] In one embodiment, the unified multimodal understanding and generation model further includes an initial generation module. The step of inputting the text prompt data and the reference image data into the unified multimodal understanding and generation module for image generation processing, and outputting target image data, includes:

[0051] The text prompt data and the reference image data are input into the display visual planning module for planning processing, and a visual inspection instruction is output.

[0052] The text prompt data and the reference image data are input into the initial generation module for image generation processing, and the initial draft image data is output.

[0053] The initial draft image data and the visual inspection instructions are input into the adaptive visual correction module for image correction processing, and the target image data is output.

[0054] In one embodiment, the visual inspection instruction includes at least one inspection item, which includes visual elements to be aligned, visual type, and target region. The process of image correction processing performed by the adaptive visual correction module includes:

[0055] For each inspection item in the visual inspection instruction, check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item.

[0056] When inconsistencies are found in the inspection results, an editing instruction is generated to indicate the content that needs to be corrected.

[0057] The initial draft image data is corrected based on the editing instructions, and the corrected image data is used as the new initial draft image data. The process then jumps to each inspection item in the visual inspection instructions to check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item, until the iterative correction conditions are met. Finally, the current initial draft image data is output as the target image data.

[0058] In one embodiment, the training process of the unified multimodal understanding and generative model includes:

[0059] A training set is constructed, and an initial unified multimodal understanding and generation model is trained based on the sample data in the training set to obtain the first unified multimodal understanding and generation model;

[0060] The first unified multimodal understanding and generation model is subjected to reinforcement learning based on the sample data in the training set to obtain the unified multimodal understanding and generation model.

[0061] In one embodiment, the training set includes a first training set and a second training set. The sample data in the first training set includes sample text prompt data and sample reference image data, and the annotation information of the sample data includes ground truth visual inspection instructions and ground truth image data.

[0062] The sample data in the second training set includes sample text prompt data, sample reference image data, defect image data generated based on the sample text prompt data and the sample reference image data, and truth visual inspection instructions. The annotation information of the sample data includes truth editing instructions and truth image data.

[0063] In one embodiment, training the initial unified multimodal understanding and generation model based on sample data in the training set includes:

[0064] Randomly sample data from the first training set and the second training set;

[0065] The sampled data is input into the initial unified multimodal understanding and generation model for image generation processing, and the output is the predicted information;

[0066] The prediction loss is determined based on the prediction information and the annotation information of the sample data, and the parameters of the initial unified multimodal understanding and generation model are tuned based on the prediction loss.

[0067] In one embodiment, when the sample data comes from the first training set, the prediction information includes predicted visual inspection instructions and predicted image data, and determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0068] The first loss is determined based on the predicted visual inspection instruction and the true visual inspection instruction corresponding to the sample data.

[0069] The second loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0070] The predicted loss is determined based on the first loss and the second loss.

[0071] In one embodiment, when the sample data comes from the second training set, the prediction information includes predicted editing instructions and predicted image data, and determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0072] The third loss is determined based on the prediction editing instructions and the truth value editing instructions corresponding to the sample data;

[0073] The fourth loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0074] The predicted loss is determined based on the third loss and the fourth loss.

[0075] In one embodiment, the reinforcement learning of the first unified multimodal understanding and generation model based on sample data in the training set includes:

[0076] The sample text prompt data and sample reference image data in the sample data are input into the first unified multimodal understanding and generation model for image generation processing, and the output is the predicted visual inspection instruction and the predicted image data.

[0077] The target reward value is determined based on the predicted visual inspection instructions, the predicted image data, the sample text prompt data, and the sample reference image data;

[0078] The target reward value is used as the reward signal for reinforcement learning, and the first unified multimodal understanding and generation model is optimized based on the reward signal.

[0079] In one embodiment, determining the target reward value based on the predicted visual inspection instruction and the predicted image data, as well as the sample text prompt data and the sample reference image data, includes:

[0080] A first reward value is determined based on the predicted image data and the sample text prompt data, wherein the first reward value is used to characterize the textual semantic consistency between the predicted image data and the sample text prompt data;

[0081] A second reward value is determined based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data. The second reward value is used to characterize the visual feature consistency between the predicted image data and the sample reference image data.

[0082] The target reward value is determined based on the first reward value and the second reward value.

[0083] In one embodiment, determining the second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data includes:

[0084] For any inspection item in the predicted visual inspection instruction, a reward value measurement model corresponding to the inspection item is determined according to the visual type indicated by the inspection item, and the visual feature consistency between the predicted image data and the sample reference image data is determined using the reward value measurement model to obtain the reward value corresponding to the inspection item.

[0085] The second reward value is determined based on the reward value corresponding to each of the aforementioned inspection items.

[0086] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the image generation methods provided in the first aspect.

[0087] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the image generation methods provided in the first aspect.

[0088] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform any of the image generation methods provided in the first aspect.

[0089] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0090] The image generation method, apparatus, electronic device, and storage medium provided in this disclosure acquire text prompt data and reference image data, input the text prompt data and reference image data into a unified multimodal understanding and generation model for image generation processing, and output target image data. The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module generates visual inspection instructions based on the text prompt data and reference image data. The visual inspection instructions guide the adaptive visual correction module to adaptively correct the currently generated image data, and finally output the target image data. By introducing explicit visual planning and adaptive visual correction mechanisms, the image generation method, apparatus, electronic device, and storage medium provided in this disclosure enable the model to actively identify, plan, critique, and correct parts of the generated image data that are inconsistent with the visual context based on the text prompt data and reference image data. This achieves high-fidelity, high-consistency, and interpretable precise control over core visual elements and the overall scene composition, thereby significantly improving the visual quality and controllability of the generated image content.

[0091] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0092] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0093] Figure 1 is a flowchart illustrating an image generation method according to an exemplary embodiment.

[0094] Figure 2 is a detailed flowchart illustrating step 104 according to an exemplary embodiment.

[0095] Figure 3 is a detailed flowchart illustrating an image correction process according to an exemplary embodiment.

[0096] Figure 4 is a detailed flowchart illustrating the model training process according to an exemplary embodiment.

[0097] Figure 5 is a detailed flowchart illustrating step 402 according to an exemplary embodiment.

[0098] Figure 6 is a detailed flowchart illustrating step 506 according to an exemplary embodiment.

[0099] Figure 7 is a detailed flowchart illustrating step 506 according to another exemplary embodiment.

[0100] Figure 8 is a detailed flowchart illustrating step 404 according to an exemplary embodiment.

[0101] Figure 9 is a detailed flowchart illustrating step 804 according to an exemplary embodiment.

[0102] Figure 10 is a detailed flowchart illustrating step 904 according to an exemplary embodiment.

[0103] Figure 11 is a block diagram illustrating the principle of an image generation method according to an exemplary embodiment.

[0104] Figure 12 is a block diagram illustrating an image generation apparatus according to an exemplary embodiment.

[0105] Figure 13 is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0106] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0107] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0108] Figure 1 is a flowchart illustrating an image generation method according to an exemplary embodiment. This embodiment uses the application of the method to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 102 to 104, wherein:

[0109] Step 102: Obtain text prompt data and reference image data.

[0110] In this embodiment, the text prompt data includes user instructions to clarify the image generation intent, which limit the content, scene, style, or editing requirements of the generated image. These instructions can be flexibly expressed according to the actual application scenario, including but not limited to scene descriptions, style requirements, and editing instructions. Reference image data serves as the benchmark for the model to maintain visual context consistency and is a key component of the input visual context. It includes, but is not limited to, various visual conditional data such as character identity reference images, scene layout reference images, pose skeleton images, semantic segmentation images, and edge images. Reference image data is used to provide key visual features that need to be retained or aligned.

[0111] The acquisition of text prompt data and reference image data can be achieved through methods such as local upload on the terminal, retrieval from a server database, synchronization via third-party platform interfaces, and real-time shooting. For example, taking a multi-reference image synthesis scenario, the text prompt data consists of merging the person in Figure 1 with the city night scene in Figure 2, preserving the person's identity and the night scene style. The reference image data includes Figure 1 (used to provide a baseline for the person's identity) and Figure 2 (used to provide a baseline for the night scene style and layout). The text prompt data and reference image data together constitute the input to the model.

[0112] It should be noted that the embodiments disclosed herein do not impose specific limitations on the form of the text prompt data (such as short sentences, long sentences, keyword combinations) or the type and number of reference image data. These limitations can be flexibly adjusted according to the complexity of the image generation task (such as single image generation, multi-frame storyboard generation, image editing, multi-reference fusion), as long as the generation intent and visual alignment benchmark can be clearly conveyed.

[0113] Step 104: Input the text prompt data and reference image data into the unified multimodal understanding and generation model for image generation processing, and output the target image data. The unified multimodal understanding and generation model includes a display vision planning module and an adaptive vision correction module. The display vision planning module is used to generate visual inspection instructions based on the text prompt data and reference image data. The visual inspection instructions are used to guide the adaptive vision correction module to adaptively correct the currently generated image data and finally output the target image data.

[0114] In this embodiment, the unified multimodal understanding and generation model is a fusion model designed based on a closed-loop logic of planning, generation, and correction. Through the collaborative work of an explicit visual planning module and an adaptive visual correction module, it addresses the problem of insufficient visual context consistency in traditional models, ensuring that the generated image not only meets the requirements of the text prompt data but also maintains consistency with the key visual features of the reference image data. For example, the unified multimodal understanding and generation model can first generate visual inspection instructions through the explicit visual planning module, clarifying visual alignment rules through these instructions. Then, it generates initial draft image data based on the text prompt data and reference image data. Finally, the adaptive visual correction module iteratively optimizes the defects of the initial draft image data based on the visual alignment rules, ultimately outputting target image data that satisfies visual context consistency.

[0115] The explicit visual planning module generates structured visual inspection instructions, which translate the vague requirements for visual context consistency into specific rules that the model can execute. After receiving text prompt data and reference image data, the explicit visual planning module automatically parses the core information from both, clarifies the alignment required for the generated image to maintain visual context consistency, and generates corresponding visual inspection instructions.

[0116] The adaptive visual correction module can check and iteratively optimize the images during the generation process based on visual inspection instructions, ensuring that all visual alignment rules indicated by the visual inspection instructions are met. For example, if the visual inspection instructions output by the display visual planning module indicate that the elderly person's coat in the image needs to maintain visual context consistency with the coat in the reference image data, then if the color of the elderly person's coat in the initially generated image is black (which does not match the gray in the reference image data), the adaptive visual correction module can identify this attribute alignment deviation and correct the color of the elderly person's coat to gray during the iterative optimization process to ensure that the coat color is consistent with that in the reference image data.

[0117] It should be noted that the specific parameters of the unified multimodal understanding and generation model are not specifically limited in the embodiments of this disclosure, such as the generation accuracy of visual inspection instructions, the threshold of the number of iterations, the specific implementation method of the correction algorithm, the subdivision dimension of the alignment type, etc. Those skilled in the art can flexibly configure them according to the accuracy requirements of the generation task, the data scale, hardware performance and other information.

[0118] The image generation method provided in this disclosure acquires text prompt data and reference image data, inputs the text prompt data and reference image data into a unified multimodal understanding and generation model for image generation processing, and outputs target image data. The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module generates visual inspection instructions based on the text prompt data and the reference image data. The visual inspection instructions guide the adaptive visual correction module to adaptively correct the currently generated image data, and finally output the target image data. By introducing explicit visual planning and adaptive visual correction mechanisms, the image generation method provided in this disclosure enables the model to actively identify, plan, critique, and correct parts of the generated image data that are inconsistent with the visual context based on the text prompt data and reference image data. This achieves high-fidelity, high-consistency, and interpretable precise control over core visual elements and the overall scene composition, thereby significantly improving the visual quality and controllability of the generated image content.

[0119] In an exemplary embodiment, the unified multimodal understanding and generation model further includes an initial generation module. Referring to FIG2, in step 104, text prompt data and reference image data are input into the unified multimodal understanding and generation module for image generation processing, and the target image data is output. This may include steps 202 to 206, wherein:

[0120] Step 202: Input the text prompt data and reference image data into the visual planning module for planning processing, and output the visual inspection instructions;

[0121] Step 204: Input the text prompt data and reference image data into the initial generation module for image generation processing, and output the initial draft image data;

[0122] Step 206: Input the initial draft image data and visual inspection instructions into the adaptive visual correction module for image correction processing, and output the target image data.

[0123] In this embodiment, after receiving text prompt data and reference image data, the explicit visual planning module first parses both, including extracting the generation intent (such as scene, style, editing requirements) from the text prompt data and identifying key visual features (such as people, objects, spatial layout, style tone) from the reference image data. Further, the explicit visual planning module generates structured visual inspection instructions based on the parsing results.

[0124] The visual inspection instructions provide a basis for visual alignment in subsequent image generation and correction. They quantify visual consistency requirements through inspection items and include at least one inspection item. Each inspection item is the smallest verifiable and executable unit. By combining the visual elements, visual types, and target regions that need to be aligned, it defines what to align, by what dimension, and within which range of the image, ensuring that the model can explicitly perform the visual alignment operation.

[0125] The visual elements that need to be aligned refer to the visual content that must maintain consistency with the reference image data in the generated image data. They are the objects of visual alignment and are generated based on key visual features in the reference image data. For example, visual elements can be flexibly selected according to the requirements of the generation task, including but not limited to elements related to people (such as facial features, hairstyle, and body shape), elements related to objects (such as the shape of a table, the color of a cup, and the cover pattern of a book), elements related to style (such as brushstrokes in an oil painting, the effect of ink wash painting, and the line style of a cartoon), and elements related to space (such as the layering and occlusion relationships between objects).

[0126] For example, if the reference image data is: a red ceramic cup placed on a wooden table, the visual elements that need to be aligned may include: the red ceramic cup (the object itself), the wooden table (the object itself), the red color of the cup (object attribute feature), the wood texture of the table (object attribute feature), etc.

[0127] The visual type requiring alignment refers to the specific dimensions or standards of the alignment operation. It clarifies the rules for aligning visual elements to ensure accurate alignment direction. Visual types are set around key dimensions of visual context consistency, including but not limited to identity alignment (unique identifiers of people or objects, such as facial features or customized logos), style alignment (the overall artistic tone of an image, such as realistic, abstract, retro, or sci-fi), attribute alignment (the inherent characteristics of an object, such as color, material, size, and state), and spatial relationship alignment (the positional relationships between multiple elements, such as front and back, left and right, top and bottom, occlusion, and proportion).

[0128] For example, taking the reference image data from the previous example, the visual type corresponding to the red ceramic cup may include: attribute alignment (color: red, material: ceramic) and identity alignment (the unique shape of the cup). The visual type corresponding to the positional relationship between the cup and the table is: spatial relationship alignment (the cup is above the table).

[0129] The target area to be aligned refers to the applicable scope of the visual alignment rules. It is used to clarify where to perform the alignment operation in the generated image data, avoiding deviations caused by confusion in the alignment range. The target area can be flexibly defined according to the distribution of visual elements, including but not limited to the global area (suitable for style alignment and other scenarios that need to cover the entire image), the local area (suitable for the alignment of specific objects or features, such as the cup area on the left side of the image, the desktop area in the lower right corner of the image, etc.), and the precise coordinate area (suitable for high-precision alignment scenarios, such as specific areas located by pixel coordinates).

[0130] For example, taking the reference image data from the previous example, the target area of ​​the red ceramic cup can be set to the area where the cup is located in the upper part of the image, the target area of ​​the wooden tabletop can be set to the area where the tabletop is located in the lower part of the image, and the target area for style alignment can be set to the entire image.

[0131] It should be noted that the embodiments disclosed herein do not impose specific limitations on the number of inspection items, the specific types of visual elements, the subdivision dimensions of visual types, or the definition method of target regions. In practical applications, inspection items can be flexibly added or removed, visual type dimensions adjusted, and target region definition methods optimized according to the complexity of the generation task (such as single object generation, multi-element fusion, and multi-frame coherent generation) and accuracy requirements, ensuring that each inspection item can clearly and unambiguously guide the model to complete the visual context alignment operation.

[0132] By constructing inspection items from visual elements, visual types, and target regions, visual inspection instructions quantify the visual context alignment requirements, enabling the model to have clear execution standards during image generation and correction. This effectively avoids the problem of insufficient visual context consistency caused by ambiguous alignment requirements in traditional models, and improves the quality of the generated image data.

[0133] The initial generation module's function is to quickly generate initial draft image data that fits the requirements based on the input text prompt data and reference image data. In other words, the initial generation module balances the need to realize the text intent with the initial adaptation of reference features. On the one hand, it strictly adheres to the scene, content, and style requirements in the text prompt data; on the other hand, it initially extracts basic visual features from the reference image data (such as the shape of main objects and overall color tone) and incorporates them into the generation process to generate initial draft image data, avoiding excessive deviation between the initial draft image data and the reference image data.

[0134] It should be noted that the generation order of the visual inspection instruction and the initial draft image data is not specifically limited in this embodiment. Step 202 can be executed first and then step 204 can be executed, or step 204 can be executed first and then step 202 can be executed, or steps 202 and step 204 can be executed simultaneously.

[0135] After generating visual inspection instructions and initial draft image data, the adaptive visual correction module can optimize the initial draft image into target image data that meets all alignment requirements, based on the guidance of the visual inspection instructions. For example, the adaptive visual correction module can check the initial draft image data against each inspection item in the visual inspection instructions, identify deviations that do not meet the alignment requirements represented by the inspection items, and perform local optimization on the deviations without altering the parts of the image data that already meet the requirements. For instance, for a sofa color deviation, only the hue of the sofa area on the left side of the image is adjusted to beige; for a chandelier position deviation, the chandelier's coordinates are fine-tuned to be directly above the sofa.

[0136] In this way, the embodiments of this disclosure, by clearly defining alignment rules through visual inspection instructions, can avoid deviations in image generation direction. Furthermore, the initial draft image data generated by the initial generation module enables rapid construction of the basic framework, ensuring efficient image generation. The adaptive visual correction module, combined with visual inspection instructions, iteratively optimizes the details of the initial draft image data, ensuring consistency of the visual context. The entire process balances image generation speed and quality, and through structured instructions and iterative correction operations, it addresses the problems of inaccurate visual feature context alignment and insufficient cross-scene coherence in traditional models, further improving the quality of the generated image data.

[0137] In an exemplary embodiment, referring to FIG3, the process of the adaptive visual correction module performing image correction processing may include the following steps 302 to 306, wherein:

[0138] Step 302: For each inspection item in the visual inspection instruction, check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual elements and visual type specified by the inspection item.

[0139] Step 304: When there is an inconsistency in the inspection results, an editing instruction is generated to indicate the content that needs to be corrected.

[0140] Step 306: Correct the initial draft image data based on the editing instructions, and use the corrected image data as the new initial draft image data. Jump to each inspection item in the visual inspection instructions to check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual elements and visual types specified by the inspection item, until the iterative correction conditions are met, and output the current initial draft image data as the target image data.

[0141] In this embodiment of the disclosure, the adaptive visual correction module can parse all inspection items in the visual inspection instruction and initiate the verification process one by one in sequence. For example, it can first locate the target area specified by the inspection item, then extract the image content features (such as color, shape, style, and spatial location) within the target area, and finally match and verify the extracted features with the visual elements and visual types specified by the inspection item.

[0142] For example, if the visual type is identity alignment, then the identity features (such as facial features) of the visual elements in the target area of ​​the initial draft image are checked to see if they match the corresponding visual elements in the reference image data; if it is attribute alignment, then the attribute features (such as color, material, size) of the visual elements in the target area of ​​the initial draft image are checked to see if they match the attributes of the corresponding visual elements in the reference image data; if it is spatial relationship alignment, then the positional relationship of multiple visual elements in the target area of ​​the initial draft image is checked to see if it matches the spatial relationship of the corresponding visual elements in the reference image data; if it is style alignment, then the style features of the target area of ​​the initial draft image are checked to see if they match the style features of the visual elements in the reference image data.

[0143] For example, if a certain inspection item is (visual element: red ceramic material, visual type: attribute alignment, target area: upper cup area in the image), the adaptive visual correction module will locate the upper cup area in the initial draft image data, extract the color (e.g., pink) and material (e.g., plastic texture) features of the cup in that area, match them with the red ceramic in the reference image data, and finally obtain the inspection result that there is a double inconsistency in color and material.

[0144] It should be noted that the specific technical means of verification (such as feature extraction algorithms and matching thresholds) are not limited in the embodiments disclosed herein. Those skilled in the art can select appropriate visual feature matching schemes according to accuracy requirements.

[0145] After completing the inspection based on visual inspection instructions and obtaining the inspection results, if inconsistencies exist in the inspection results, the adaptive visual correction module will automatically generate structured editing instructions based on the inconsistencies. The editing instructions can include information such as the location of the inconsistency (i.e., the target area specified by the inspection item), the type of inconsistency (e.g., color deviation, material deviation, spatial position deviation, etc.), and the correction target (e.g., correcting the color to red, correcting the material to a ceramic texture, etc.). For example, using the above example again, for the inconsistency of the cup's pink color and plastic texture, the generated editing instructions could be: correct the color of the upper cup area in the image to red, and optimize the material to a ceramic glossy texture.

[0146] It should be noted that the specific format and expression of the editing instructions are not limited in the embodiments disclosed herein. Any method that can directly guide the model to make targeted corrections is applicable to the embodiments of this application.

[0147] After the adaptive visual correction module generates the editing instructions, it only performs local optimization on the target area specified in the editing instructions, without modifying other eligible areas in the image data. For example, based on the instruction to correct the cup color to red, it only adjusts the tone channel of the cup area, keeping the features of other areas such as the desktop and background unchanged. After correction is completed, the newly generated image data is redefined as the initial draft image data, and it automatically jumps back to step 302 for iterative optimization, starting a new round of inspection and correction operations (that is, re-verifying all inspection items in the visual inspection instructions).

[0148] The iterative process continues until preset iterative correction conditions are met. These conditions can include: all check items meeting alignment requirements (i.e., the image content in the target region specified by the check item in the initial draft image data is consistent with the visual elements and visual types specified by the check item); and reaching the maximum number of iterations (i.e., stopping iteration to balance image quality and image generation efficiency when the maximum number of iterations is reached). During the iteration process, if any of these conditions are met, the iteration flow can be stopped, and the current image data can be output as the target image data.

[0149] The image generation method provided in this disclosure performs iterative optimization by constraining visual type, visual elements, and target region through check items. By clearly defining the alignment object, dimension, and range through check items, the accuracy of visual alignment can be greatly improved. Furthermore, the iterative optimization can take into account the consistency of multiple dimensions such as identity, attributes, style, and space, ensuring that the final generated image data meets the requirements of visual context consistency, which can greatly improve the quality of the generated image data.

[0150] In an exemplary embodiment, referring to FIG4, the training process of the unified multimodal understanding and generative model may include the following steps 402 to 404, wherein:

[0151] Step 402: Construct a training set and train an initial unified multimodal understanding and generation model based on the sample data in the training set to obtain the first unified multimodal understanding and generation model;

[0152] Step 404: Reinforcement learning is performed on the first unified multimodal understanding and generation model based on the sample data in the training set to obtain the unified multimodal understanding and generation model.

[0153] In this embodiment of the disclosure, a training set for training a unified multimodal understanding and generation model can be pre-constructed, and an initial unified multimodal understanding and generation model can be trained based on the sample data in the training set. After training, a first unified multimodal understanding and generation model is obtained, which is a version to be reinforced learning.

[0154] In an exemplary embodiment, the training set includes a first training set and a second training set. The sample data in the first training set includes sample text prompt data and sample reference image data, and the annotation information of the sample data includes ground truth visual inspection instructions and ground truth image data. The sample data in the second training set includes sample text prompt data, sample reference image data, defect image data generated based on the sample text prompt data and sample reference image data, and ground truth visual inspection instructions, and the annotation information of the sample data includes ground truth editing instructions and ground truth image data.

[0155] In this embodiment of the disclosure, the training set can be split into a first training set and a second training set, which are used to specifically train the model's basic generation and visual planning capabilities, and deviation detection and editing correction capabilities, thereby achieving hierarchical and refined training of the model's capabilities.

[0156] The first training set is used to train the model's visual planning and basic generation capabilities. The sample data in the first training set includes sample text prompts and sample reference images, and the annotation information for the sample data includes ground truth visual inspection instructions and ground truth image data. Specifically, the sample text prompts correspond to the user's image generation intent instructions in real-world applications, and the sample reference images correspond to the visual benchmarks required for image generation. Together, they simulate the input format of user text instructions plus visual references in real-world scenarios.

[0157] The annotation information, also known as ground truth data, can include ground truth visual inspection instructions and ground truth image data. The ground truth visual inspection instructions are structured instructions generated by a pre-trained teacher VLM (Vision-Language Model) based on sample text prompt data, sample reference image data, and ground truth image data. These instructions specify the visual elements, visual types, and alignment rules for the target region that must be met to generate a compliant image. The ground truth image data is the ideal generated result that perfectly matches the requirements of the text prompt data and is consistent with the visual context of the reference image data.

[0158] The second training set is used to train the model's adaptive visual correction capabilities. The sample data in the second training set includes sample text prompts, sample reference images, defective image data generated based on the sample text prompts and reference images, and ground truth visual inspection instructions. The annotation information for the sample data includes ground truth editing instructions and ground truth image data. In other words, the sample data in the second training set is based on the sample data in the first training set, with the addition of two types of data: defective image data and ground truth visual inspection instructions. The defective image data is generated by the baseline model (an untrained initial model or a weak base generative model) based on the text prompts and reference images, and it contains various typical visual biases, simulating defects in the initial draft image data that may occur in actual model generation. The ground truth visual inspection instructions are consistent with the ground truth visual inspection instructions in the annotation information of the sample data in the first training set; they are simply transferred from the annotation data to the sample data.

[0159] The annotation information for the sample data in the second training set may include truth value editing instructions and truth value image data, wherein the truth value image data is consistent with the truth value image data in the annotation information of the sample data in the first training set. The truth value editing instructions are structured correction instructions generated by the teacher's VLM based on the truth value visual inspection instructions, defect image data, and truth value image data, and are used to specify the area to be corrected, the type of deviation, and the correction target.

[0160] By splitting the training set into a first and a second set, the model's basic generation and visual planning capabilities, as well as its bias detection and editing correction capabilities, can be trained separately. This avoids the problem of the model's capabilities being unbalanced due to a single training set and improves the model's image generation accuracy.

[0161] In an exemplary embodiment, referring to FIG5, step 402, training an initial unified multimodal understanding and generation model based on sample data in the training set, may include steps 502 to 506, wherein:

[0162] Step 502: Randomly sample data from the first training set and the second training set;

[0163] Step 504: Input the sampled data into the initial unified multimodal understanding and generation model for image generation processing, and output the prediction information.

[0164] Step 506: Determine the prediction loss based on the prediction information and the annotation information of the sample data, and tune the parameters of the initial unified multimodal understanding and generation model based on the prediction loss.

[0165] In this embodiment, sample data can be randomly sampled from the first and second training sets to train the initial unified multimodal understanding and generation model. During the sampling process from the first and second training sets, a random and uniform principle can be followed, or the sampling ratio of the two training sets can be adjusted according to the model training focus (e.g., 1:1 mixed sampling, or increasing the sampling ratio of the second training set when focusing on correction capabilities). This embodiment does not specifically limit the specific sampling ratio or the number of samples in a single sampling; those skilled in the art can set these according to training needs. After obtaining the sample data, it can be input into the initial unified multimodal understanding and generation model for image generation processing. Based on the generated prediction information and the annotation information of the sample data, the prediction loss is calculated, and based on the prediction loss, a gradient descent-type algorithm is used to backpropagate and update the parameters of the initial unified multimodal understanding and generation model.

[0166] The image generation method provided in this disclosure can simultaneously learn planning and correction capabilities by cross-training the first training set and the second training set, avoiding the problem of model bias. Furthermore, the unified iterative parameter tuning during the training process can avoid parameter conflicts and greatly shorten the training cycle.

[0167] In an exemplary embodiment, referring to FIG6, when the sample data comes from the first training set, the prediction information includes predicted visual inspection instructions and predicted image data. In step 506, determining the prediction loss based on the prediction information and the annotation information of the sample data may include the following steps 602 to 606, wherein:

[0168] Step 602: Determine the first loss based on the predicted visual inspection instructions and the true visual inspection instructions corresponding to the sample data;

[0169] Step 604: Determine the second loss based on the ground truth image data corresponding to the predicted image data and the sample data;

[0170] Step 606: Determine the predicted loss based on the first loss and the second loss.

[0171] In this embodiment of the disclosure, when the sample data comes from the first training set, it includes sample text prompt data and sample reference image data. When these are input into the initial unified multimodal understanding and generation model, the output prediction information may include predicted visual inspection instructions and predicted image data.

[0172] The first loss can be determined by comparing the predicted visual inspection instructions with the ground truth visual inspection instructions in the annotation information of the sample data, and the second loss can be determined by comparing the predicted image data with the ground truth image data in the annotation information of the sample data. The process of determining the first loss may include comparing the predicted visual inspection instructions (containing structured content such as visual elements, visual types, and target regions) with the ground truth visual inspection instructions field-by-field to quantify the deviations in alignment objects, alignment dimensions, and alignment ranges. The process of determining the second loss may include comparing the predicted image data with the ground truth image data at the pixel-level and feature-level to quantify the deviations in dimensions such as visual elements, attributes, spatial relationships, and style.

[0173] After calculating the first loss and the second loss, a weighted sum of the first loss and the second loss can be performed to obtain the final predicted loss. It should be noted that this disclosure does not limit the specific calculation functions for the first loss and the second loss; any loss function capable of quantifying the difference is applicable to this application's embodiments.

[0174] The image generation method provided in this disclosure can simultaneously optimize the dual-dimensional capabilities of model visual planning and basic generation by separately quantifying the first loss generated by visual inspection instructions and the second loss generated by image data. This avoids the model capability imbalance problem caused by single-dimensional loss. Furthermore, during the loss calculation process, a unified prediction loss can be determined by weighted integration, which can ensure that the model parameter tuning objectives are consistent and avoid conflicts in multi-dimensional optimization directions. This can greatly improve the capability optimization efficiency in the basic training stage of the model.

[0175] In an exemplary embodiment, referring to FIG7, when the sample data comes from the second training set, the prediction information includes prediction editing instructions and prediction image data. In step 506, determining the prediction loss based on the prediction information and the annotation information of the sample data may include the following steps 702 to 706, wherein:

[0176] Step 702: Determine the third loss based on the prediction editing instructions and the truth editing instructions corresponding to the sample data;

[0177] Step 704: Determine the fourth loss based on the ground truth image data corresponding to the predicted image data and the sample data;

[0178] Step 706: Determine the predicted loss based on the third and fourth losses.

[0179] In this embodiment of the application, when the sample data comes from the second training set, it includes sample text prompt data, sample reference image data, defect image data, and ground truth visual inspection instructions. When these are input into the initial unified multimodal understanding and generation model, the output prediction information may include prediction editing instructions and prediction image data.

[0180] A third loss can be determined by predicting editing instructions and ground truth editing instructions in the annotation information of sample data, and a fourth loss can be determined by predicting image data and ground truth image data in the annotation information of sample data. The process of determining the third loss may include comparing the predicted editing instructions with the ground truth visual inspection instructions field by field to quantify their deviations in the correction area, deviation type, and correction target. The process of determining the fourth loss can refer to the calculation process of the second loss, and will not be elaborated further in this embodiment.

[0181] After calculating the third and fourth losses, a weighted sum of the third and fourth losses can be performed to obtain the final predicted loss. It should be noted that this disclosure does not limit the specific calculation functions for the third and fourth losses; any loss function capable of quantifying the difference is applicable to this application.

[0182] The image generation method provided in this disclosure can simultaneously optimize the dual-dimensional capabilities of model bias checking, correction, and basic generation by quantifying the third loss generated by editing instructions and the fourth loss generated by image data, thereby avoiding the model capability imbalance caused by single-dimensional loss. Furthermore, during the loss calculation process, a unified prediction loss can be determined through weighted integration, which can ensure that the model parameter tuning objectives are consistent and avoid conflicts in multi-dimensional optimization directions, thereby greatly improving the capability optimization efficiency in the basic training stage of the model.

[0183] After training the first unified multimodal understanding and generation model according to the training process of the aforementioned embodiment, reinforcement learning can be performed on the first unified multimodal understanding and generation model to finally obtain the unified multimodal understanding and generation model.

[0184] The image generation method provided in this disclosure, through two-stage training of basic training and reinforcement learning, can first solidify the basic capabilities of model planning, generation and correction, and then optimize core performance. This avoids the problem of weak model capabilities or insufficient generalization caused by a single training stage. Furthermore, during the training process, reinforcement learning is used to tune parameters end-to-end with reward signals as a guide, which can accurately optimize the model's visual consistency and textual fit. This avoids the slow performance improvement caused by parameter tuning without a target, and can greatly improve the final image generation accuracy and reliability of the model.

[0185] In an exemplary embodiment, referring to FIG8, step 404, performing reinforcement learning on the first unified multimodal understanding and generation model based on sample data in the training set, may include steps 802 to 806, wherein:

[0186] Step 802: Input the sample text prompt data and sample reference image data from the sample data into the first unified multimodal understanding and generation model for image generation processing, and output the predicted visual inspection instructions and predicted image data.

[0187] Step 804: Determine the target reward value based on the predicted visual inspection instructions and predicted image data, as well as the sample text prompt data and sample reference image data;

[0188] Step 806: Use the target reward value as the reward signal for reinforcement learning, and optimize the first unified multimodal understanding and generation model based on the reward signal.

[0189] In this embodiment of the present disclosure, during the reinforcement learning phase, the sample text prompt data and sample reference image data in the sample data can be output to the first unified multimodal understanding and generation model for image generation processing, and the predicted visual inspection instructions and predicted image data can be output. The specific process is described in the relevant description of the foregoing embodiments, and will not be repeated here in this embodiment of the present disclosure.

[0190] Furthermore, the target reward value can be determined based on the predicted visual inspection instructions and predicted image data, as well as the sample text prompt data and sample reference image data. For example, the target reward value can be calculated using a composite reward function, which can quantify both visual contextual consistency and text semantic consistency separately, determining the final target reward value through the quantified values ​​of consistency across these two dimensions.

[0191] In an exemplary embodiment, referring to FIG9, step 804, determining the target reward value based on the predicted visual inspection instructions and predicted image data, as well as the sample text prompt data and sample reference image data, may include steps 902 to 906, wherein:

[0192] Step 902: Determine a first reward value based on the predicted image data and the sample text prompt data. The first reward value is used to characterize the text semantic consistency between the predicted image data and the sample text prompt data.

[0193] Step 904: Determine a second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data. The second reward value is used to characterize the visual feature consistency between the predicted image data and the sample reference image data.

[0194] Step 906: Determine the target reward value based on the first reward value and the second reward value.

[0195] In this embodiment, the predicted image data can be converted into standardized visual feature vectors, and the sample text prompt data can be converted into text feature vectors. A first reward value is then calculated by determining the cosine similarity between the two types of vectors, with a value ranging from [0,1]. The closer the value is to 1, the higher the semantic consistency between the predicted image data and the sample text prompt data. For example, if the sample text prompt data is: "A white ceramic bowl is placed on a wooden plate in the reference image," and the predicted image data also shows a white ceramic bowl placed on a wooden plate, then the matching degree between the two feature vectors is high, and the first reward value is high. If the predicted image data shows a black plastic bowl placed on a metal plate, then the matching degree between the two feature vectors is low, and the first reward value is low.

[0196] It should be noted that the specific calculation algorithm for the first reward value is not limited in the embodiments of this disclosure. Any method that can effectively quantify the degree of semantic matching between the predicted image data and the text prompt data is applicable to the embodiments of this disclosure.

[0197] Furthermore, all inspection items in the predicted visual inspection instructions can be analyzed to extract visual elements, visual types, and target regions. Then, for different visual types, the corresponding objective visual measurement models (such as Face-ID similarity for identity alignment, feature cosine similarity for attribute alignment, and pixel-level positional deviation algorithm for spatial relationship alignment) are called to quantify the consistency between the visual features in the target region of the predicted image data and the corresponding region features of the sample reference image data. Finally, the quantification results corresponding to each inspection item are weighted and summed to obtain the second reward value, which ranges from [0,1]. The closer the value is to 1, the higher the consistency of the visual features between the predicted image data and the sample reference image data.

[0198] In an exemplary embodiment, referring to FIG10, step 904, determining the second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data, may include steps 1002 to 1004, wherein:

[0199] Step 1002: For any inspection item in the predicted visual inspection instruction, determine the reward value measurement model corresponding to the inspection item according to the visual type indicated by the inspection item, and use the reward value measurement model to determine the visual feature consistency between the predicted image data and the sample reference image data to obtain the reward value corresponding to the inspection item.

[0200] Step 1004: Determine the second reward value based on the reward value corresponding to each inspection item.

[0201] In this embodiment, each inspection item in the predicted visual inspection instruction can be parsed and its visual type (such as identity alignment, attribute alignment, spatial relationship alignment, and style alignment) can be extracted. Then, based on the feature dimension of the visual type, a corresponding reward value measurement model is matched. For example, identity alignment (such as facial features) is adapted to the Face-ID similarity model, attribute alignment (such as inherent features such as color, material, and size) is adapted to the feature cosine similarity model, spatial relationship alignment (such as positional associations between objects such as front and back, left and right, and occlusion) is adapted to the pixel-level positional deviation calculation model, and style alignment (such as artistic tones such as oil painting brushstrokes, ink wash, and cartoon lines) is adapted to the style similarity model. Finally, for the target area specified by the inspection item, the matched reward value measurement model is called to quantify the feature consistency between the predicted image data and the sample reference image data under the visual type, and the reward value corresponding to a single inspection item is output.

[0202] For example, if a certain inspection item is (visual element: black leather sofa, visual type: attribute alignment, target area: sofa area), a feature cosine similarity model is matched based on the visual type to extract the color and material feature vectors of the sofa in the central living room area of ​​the image from the predicted image data and the sample reference image data, and the vector similarity is calculated to obtain the reward value corresponding to the inspection item; if the inspection item is (visual element: sofa and coffee table center alignment, visual type: spatial relationship alignment, target area: center of the image), a pixel-level position deviation model is matched to calculate the position deviation between the sofa and coffee table in the predicted image data and convert it into the corresponding reward value.

[0203] Furthermore, by weighted summing the reward values ​​corresponding to each inspection item, a second reward value can be obtained. For example, if the predicted visual inspection instruction contains three inspection items: inspection item 1 (reward value 0.8, weight 0.4), inspection item 2 (reward value 0.7, weight 0.4), and inspection item 3 (reward value 0.9, weight 0.2), then the second reward value = 0.4 × 0.8 + 0.4 × 0.7 + 0.2 × 0.9 = 0.78.

[0204] The second reward value determination method provided in this disclosure quantifies the reward value of a single inspection item by matching a dedicated metric model according to visual type. This can accurately measure the consistency of visual features in each dimension, avoid quantification bias caused by generalization evaluation, and determine the second reward value by weighted integration of all inspection item reward values. This can comprehensively cover the multi-dimensional visual alignment requirements, avoid the one-sidedness of single inspection item evaluation, and greatly improve the quantification accuracy of visual dimension reward values, providing a more reliable visual alignment guide for reinforcement learning.

[0205] After calculating the first reward value and the second reward value, the target reward value can be calculated by weighted summation of the first reward value and the second reward value.

[0206] The target reward value determination method provided in this disclosure can simultaneously measure the textual fit and visual alignment of the predicted image by separately quantifying the first reward value of textual semantic consistency and the second reward value of visual feature consistency. This avoids the problem of model optimization imbalance caused by single-dimensional reward. Furthermore, by determining a unified target reward value through weighted integration during the reward calculation process, it can ensure the consistency of the optimization goals of reinforcement learning and avoid multi-dimensional orientation conflicts, which can greatly improve the performance optimization accuracy of the model in the reinforcement learning stage.

[0207] After calculating the target reward value, it can be used as a reward signal to optimize the first unified multimodal understanding and generation model. The optimization process focuses on maximizing the target reward value and employs a GRPO (Group Relative Policy Optimization) type reinforcement learning algorithm. The target reward value is used as a feedback signal to propagate back to the model: if the target reward value is low, the model will adjust relevant parameters accordingly; if the target reward value is high, the current parameter direction is retained, and the corresponding generation strategy is strengthened. Through multiple iterations, the model gradually optimizes the accuracy of visual inspection command generation and the visual alignment of image generation, ultimately outputting image data that better meets the requirements.

[0208] The reinforcement learning process provided in this embodiment can optimize the model with reward signals as the guide, avoiding the performance bottleneck caused by untargeted parameter tuning in the basic training stage. Furthermore, the reward calculation based on the decomposition of visual dimensions according to the prediction visual inspection instructions allows the model optimization to focus more on visual alignment requirements, greatly improving the visual context consistency and text semantic fit of the final generated image data, and enhancing the reliability of the model in real-world scenarios.

[0209] To enable those skilled in the art to better understand the embodiments of this disclosure, the embodiments of this disclosure are described below through specific examples.

[0210] This disclosure proposes a method for improving the visual context consistency of a unified multimodal understanding and generative model. The core of this method lies in introducing explicit visual planning and adaptive visual correction mechanisms, implemented through a two-stage hybrid training paradigm. The unified multimodal understanding and generative model includes an explicit visual planning module, an initial generation module, and an adaptive visual correction module, wherein:

[0211] Explicit visual planning module: The model receives text input ( ) and visual context ( After that, a structured visual inspection instruction is actively generated. This instruction explicitly specifies the visual elements that need to be aligned with the input visual context, the alignment type (such as identity, style, attribute, spatial relationship), and the corresponding area.

[0212] Initial generation module: Model based on , and Generate an initial draft image data ( ).

[0213] The adaptive vision correction module includes:

[0214] Inspection phase: Model based on For the currently generated image (initial draft image data) Or the image data from the previous round of corrections It performs a self-check, identifies parts that do not meet visual consistency requirements, and provides specific editing instructions. ).

[0215] Correction phase: The model corrects the current image according to the editing instructions, generating a new round of corrected images. ).

[0216] Iterative reasoning: The inspection and correction process can be performed iteratively until the model determines that all visual inspection instructions have been met, or the preset maximum number of iterations has been reached, thereby obtaining the final image. ).

[0217] To enable the model to master the above-mentioned planning, generation, critique and correction capabilities, this disclosure designs a two-stage hybrid training paradigm, which includes a supervised fine-tuning (SFT) stage and a policy optimization stage, as shown in Figure 11.

[0218] In the SFT hybrid training phase: This phase aims to enable unified multimodal understanding and generative models through a data-driven approach. It possesses both planning and inspection capabilities and the ability to self-correct. Two datasets are constructed in this embodiment. and The model is updated by mixing samples during the same training phase.

[0219] The SFT-1 dataset ( This is used to train the model to learn its planning and ideal generation capabilities. Its data construction contains tuples. . Characterizing sample text prompt data, Characterizing sample image reference data, Characterize the truth visual inspection instructions, The true image data represents the data where, It is based on a powerful teacher VLM model (any powerful multimodal model capable of understanding complex visual scenes and generating structured instructions can serve as a teacher VLM model). Automatically generated structured visual inspection instructions. These instructions are in JSON list format, with each inspection item... It includes check_type (visual type, including Identity, Style, Attribute, Spatial), source (visual element), and target (target area).

[0220] Learning objectives: Training In the given At that time, it can generate and .

[0221] SFT-2 dataset ( This is used to train the model to learn its ability to check and correct. Data construction includes tuples. It is pre-generated before the start of SFT hybrid training. Characterizing defective image data is done using a baseline model ( )based on The generated flawed draft image data will naturally contain errors due to the lack of visual consistency of the baseline model. Representing truth value editing commands, using teacher VLM input. Generate critique of , Used to describe Not satisfied (place) and will Revised to The instructions.

[0222] Learning objectives: Training In the given At that time, it can generate And the revised .

[0223] Mixed training: Simultaneously from the SFT stage and Training is performed using sampled data.

[0224] In the policy optimization (GRPO) phase: reinforcement learning is used to further optimize the model trained in the SFT phase. This is done to maximize the visual context consistency of the final output. It includes:

[0225] Iterative inference trajectory: for each input ,Model It will generate Due to training cost considerations, we will directly use... As A composite reward function is adopted. Calculate reward value :

[0226]

[0227] in, Used for evaluation and Textual semantic consistency. It is a dynamic reward that is generated by the parsing model itself. Each inspection instruction in the code is evaluated, and the corresponding objective visual metric model (such as Face-ID similarity or feature cosine similarity) is invoked to quantify the result based on the instruction type (Identity, Style, Attribute, Spatial). and The visual feature similarity or consistency between them. The GRPO algorithm is used to... To reward signals, optimize the strategy end-to-end. To maximize expected reward.

[0228] In one example, consider a multi-reference image synthesis task. The model input includes: text prompt data. A woman wearing a blue jacket, her face is a reference image. The clothing of the people in the picture is for reference. Style reference Reference image data : (A close-up of a woman's face) (A photo of a man wearing a blue jacket) (A painting by Van Gogh).

[0229] The model processing procedure is as follows:

[0230] Perform explicit visual planning: model generation This includes:

[0231] {"check_type":"Identity","source":"V1[face]","target":"output[face]"}

[0232] {"check_type":"Identity","source":"V2[jacket]","target": output[jacket]"}

[0233] {"check_type":"Style","source":"V3","target":"output"}

[0234] Initial generation: Model generation Assuming Well preserved The faces are different, but the styles of the jackets are distinctly different.

[0235] Adaptive visual correction: Inspection phase: The model is based on... The jacket in V2 has pockets, but... The jacket in the picture has no pockets. (Generate) This includes: Checking: The jacket differs from the jacket in V2; the jacket in V2 has pockets. Correcting: The jacket needs to be more similar to the one in V2; add pockets to the jacket. Correction phase: The model is based on... right Make corrections and generate Final output: It is judged to meet all conditions and is used as the final output.

[0236] The image generation method provided in this disclosure has the following advantages:

[0237] Significantly improves visual context consistency: For the first time, explicit visual planning and adaptive visual correction are integrated into the thinking process of a unified model, enabling the model to accurately identify, plan and maintain fine-grained visual features (such as face ID, object texture, style, and spatial relationships), thereby achieving visual consistency and fidelity in complex generative tasks.

[0238] Enhancing the controllability and interpretability of models: Explicit visual programming Presented in the form of structured instructions, the model's internal thought process is made transparent. Users can observe... Understanding the visual constraints the model focuses on, and the model's generation during iterative correction. It identifies problems in the generated images and provides clear editing instructions, greatly enhancing the controllability and interpretability of the generation process.

[0239] Supports complex generation with multiple references and constraints: The model can process multiple visual reference inputs and complex text instructions simultaneously, and through its internal planning mechanism, it can effectively coordinate visual information from different sources to avoid feature conflicts and loss.

[0240] Highly efficient iterative correction capability: The adaptive visual correction mechanism enables the model to self-diagnose and correct generated visual errors. Compared with manual adjustment or implicit feedback, the embodiments of this disclosure can converge to the final result that satisfies all visual consistency conditions more quickly and accurately.

[0241] It should be understood that although the steps in the flowcharts of Figures 1-11 are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in Figures 1-11 may include multiple steps or multiple stages, which are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0242] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0243] Figure 12 is a block diagram 1200 of an image generation apparatus according to an exemplary embodiment. Referring to Figure 12, the apparatus includes an acquisition unit 1202 and an image generation processing unit 1204, wherein:

[0244] The acquisition unit 1202 is configured to acquire text prompt data and reference image data;

[0245] The image generation processing unit 1204 is configured to perform image generation processing on a unified multimodal understanding and generation model by inputting text prompt data and reference image data, and output target image data.

[0246] The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module is used to generate visual inspection instructions based on text prompt data and reference image data. The visual inspection instructions are used to guide the adaptive visual correction module to adaptively correct the currently generated image data and finally output the target image data.

[0247] In an exemplary embodiment, the unified multimodal understanding and generation model further includes an initial generation module. The step of inputting the text prompt data and the reference image data into the unified multimodal understanding and generation module for image generation processing, and outputting target image data, includes:

[0248] The text prompt data and the reference image data are input into the display visual planning module for planning processing, and a visual inspection instruction is output.

[0249] The text prompt data and the reference image data are input into the initial generation module for image generation processing, and the initial draft image data is output.

[0250] The initial draft image data and the visual inspection instructions are input into the adaptive visual correction module for image correction processing, and the target image data is output.

[0251] In an exemplary embodiment, the visual inspection instruction includes at least one inspection item, which includes visual elements to be aligned, visual type, and target region. The process of image correction processing performed by the adaptive visual correction module includes:

[0252] For each inspection item in the visual inspection instruction, check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item.

[0253] When inconsistencies are found in the inspection results, an editing instruction is generated to indicate the content that needs to be corrected.

[0254] The initial draft image data is corrected based on the editing instructions, and the corrected image data is used as the new initial draft image data. The process then jumps to each inspection item in the visual inspection instructions to check whether the image content in the target area specified by the inspection item in the initial draft image data is consistent with the visual element and visual type specified by the inspection item, until the iterative correction conditions are met. Finally, the current initial draft image data is output as the target image data.

[0255] In an exemplary embodiment, the training process of the unified multimodal understanding and generative model includes:

[0256] A training set is constructed, and an initial unified multimodal understanding and generation model is trained based on the sample data in the training set to obtain the first unified multimodal understanding and generation model;

[0257] The first unified multimodal understanding and generation model is subjected to reinforcement learning based on the sample data in the training set to obtain the unified multimodal understanding and generation model.

[0258] In an exemplary embodiment, the training set includes a first training set and a second training set. The sample data in the first training set includes sample text prompt data and sample reference image data, and the annotation information of the sample data includes ground truth visual inspection instructions and ground truth image data.

[0259] The sample data in the second training set includes sample text prompt data, sample reference image data, defect image data generated based on the sample text prompt data and the sample reference image data, and truth visual inspection instructions. The annotation information of the sample data includes truth editing instructions and truth image data.

[0260] In an exemplary embodiment, training the initial unified multimodal understanding and generation model based on sample data in the training set includes:

[0261] Randomly sample data from the first training set and the second training set;

[0262] The sampled data is input into the initial unified multimodal understanding and generation model for image generation processing, and the output is the predicted information;

[0263] The prediction loss is determined based on the prediction information and the annotation information of the sample data, and the parameters of the initial unified multimodal understanding and generation model are tuned based on the prediction loss.

[0264] In an exemplary embodiment, when the sample data comes from the first training set, the prediction information includes predicted visual inspection instructions and predicted image data, and determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0265] The first loss is determined based on the predicted visual inspection instruction and the true visual inspection instruction corresponding to the sample data.

[0266] The second loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0267] The predicted loss is determined based on the first loss and the second loss.

[0268] In an exemplary embodiment, when the sample data comes from the second training set, the prediction information includes predicted editing instructions and predicted image data, and the step of determining the prediction loss based on the prediction information and the annotation information of the sample data includes:

[0269] The third loss is determined based on the prediction editing instructions and the truth value editing instructions corresponding to the sample data;

[0270] The fourth loss is determined based on the predicted image data and the ground image data corresponding to the sample data;

[0271] The predicted loss is determined based on the third loss and the fourth loss.

[0272] In an exemplary embodiment, the reinforcement learning of the first unified multimodal understanding and generation model based on sample data in the training set includes:

[0273] The sample text prompt data and sample reference image data in the sample data are input into the first unified multimodal understanding and generation model for image generation processing, and the output is the predicted visual inspection instruction and the predicted image data.

[0274] The target reward value is determined based on the predicted visual inspection instructions, the predicted image data, the sample text prompt data, and the sample reference image data;

[0275] The target reward value is used as the reward signal for reinforcement learning, and the first unified multimodal understanding and generation model is optimized based on the reward signal.

[0276] In an exemplary embodiment, determining the target reward value based on the predicted visual inspection instruction and the predicted image data, as well as the sample text prompt data and the sample reference image data, includes:

[0277] A first reward value is determined based on the predicted image data and the sample text prompt data, wherein the first reward value is used to characterize the textual semantic consistency between the predicted image data and the sample text prompt data;

[0278] A second reward value is determined based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data. The second reward value is used to characterize the visual feature consistency between the predicted image data and the sample reference image data.

[0279] The target reward value is determined based on the first reward value and the second reward value.

[0280] In an exemplary embodiment, determining the second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data includes:

[0281] For any inspection item in the predicted visual inspection instruction, a reward value measurement model corresponding to the inspection item is determined according to the visual type indicated by the inspection item, and the visual feature consistency between the predicted image data and the sample reference image data is determined using the reward value measurement model to obtain the reward value corresponding to the inspection item.

[0282] The second reward value is determined based on the reward value corresponding to each of the aforementioned inspection items.

[0283] The image generation apparatus provided in this disclosure acquires text prompt data and reference image data, inputs the text prompt data and reference image data into a unified multimodal understanding and generation model for image generation processing, and outputs target image data. The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module generates visual inspection instructions based on the text prompt data and the reference image data. The visual inspection instructions guide the adaptive visual correction module to adaptively correct the currently generated image data, and finally output the target image data. By introducing explicit visual planning and adaptive visual correction mechanisms, the image generation apparatus provided in this disclosure allows the model to actively identify, plan, critique, and correct parts of the generated content that are inconsistent with the visual context. This achieves high-fidelity, high-consistency, interpretable, and iterative precise control over core visual elements and the overall scene composition, thereby significantly improving the visual quality and controllability of the generated image content.

[0284] Figure 13 is a block diagram illustrating an electronic device 1300 for an image generation method according to an exemplary embodiment. For example, the electronic device 1300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0285] Referring to FIG13, the electronic device 1300 may include one or more of the following components: processing component 1302, memory 1304, power supply component 1306, multimedia component 1308, audio component 1310, input / output (I / O) interface 1312, sensor component 1314, and communication component 1316.

[0286] Processing component 1302 typically controls the overall operation of electronic device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0287] Memory 1304 is configured to store various types of data to support the operation of electronic device 1300. Examples of such data include instructions for any application or method operating on electronic device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0288] Power supply component 1306 provides power to various components of electronic device 1300. Power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1300.

[0289] Multimedia component 1308 includes a screen that provides an output interface between the electronic device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0290] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when electronic device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0291] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0292] Sensor assembly 1314 includes one or more sensors for providing state assessments of various aspects of electronic device 1300. For example, sensor assembly 1314 may detect the on / off state of electronic device 1300, the relative positioning of components such as the display and keypad of electronic device 1300, changes in position of electronic device 1300 or its components, the presence or absence of user contact with electronic device 1300, orientation or acceleration / deceleration of device 1300, and temperature changes of electronic device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0293] Communication component 1316 is configured to facilitate wired or wireless communication between electronic device 1300 and other devices. Electronic device 1300 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 6G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0294] In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0295] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 1320 of an electronic device 1300 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0296] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor 1320 of an electronic device 1300 to perform the above-described method.

[0297] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0298] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0299] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that, The method includes: acquiring text prompt data and reference image data; inputting the text prompt data and the reference image data into a unified multimodal understanding and generation model for image generation processing, and outputting target image data; wherein, the unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module, the display visual planning module is used to generate visual inspection instructions based on the text prompt data and the reference image data, the visual inspection instructions are used to guide the adaptive visual correction module to adaptively correct the currently generated image data, and finally output the target image data.

2. The method according to claim 1, characterized in that, The unified multimodal understanding and generation model further includes an initial generation module. The step of inputting the text prompt data and the reference image data into the unified multimodal understanding and generation module for image generation processing and outputting target image data includes: inputting the text prompt data and the reference image data into the display visual planning module for planning processing and outputting visual inspection instructions; inputting the text prompt data and the reference image data into the initial generation module for image generation processing and outputting initial draft image data; and inputting the initial draft image data and the visual inspection instructions into the adaptive visual correction module for image correction processing and outputting target image data.

3. The method according to claim 2, characterized in that, The visual inspection instruction includes at least one inspection item, which includes visual elements, visual types, and target regions that need to be aligned. The adaptive visual correction module performs image correction processing as follows: for each inspection item in the visual inspection instruction, it checks whether the image content in the target region specified by the inspection item in the initial draft image data is consistent with the visual elements and visual types specified by the inspection item; when the inspection results indicate inconsistencies, it generates an editing instruction, which indicates the content that needs to be corrected; it corrects the initial draft image data based on the editing instruction, and uses the corrected image data as new initial draft image data, then jumps to the step of checking whether the image content in the target region specified by the inspection item in the initial draft image data is consistent with the visual elements and visual types specified by the inspection item for each inspection item in the visual inspection instruction, until the iterative correction conditions are met, and outputs the current initial draft image data as target image data.

4. The method according to any one of claims 1 to 3, characterized in that, The training process of the unified multimodal understanding and generation model includes: constructing a training set, and training an initial unified multimodal understanding and generation model based on the sample data in the training set to obtain a first unified multimodal understanding and generation model; and performing reinforcement learning on the first unified multimodal understanding and generation model based on the sample data in the training set to obtain the unified multimodal understanding and generation model.

5. The method according to claim 4, characterized in that, The training set includes a first training set and a second training set. The sample data in the first training set includes sample text prompt data and sample reference image data, and the annotation information of the sample data includes truth visual inspection instructions and truth image data. The sample data in the second training set includes sample text prompt data, sample reference image data, defect image data generated based on the sample text prompt data and the sample reference image data, and truth visual inspection instructions, and the annotation information of the sample data includes truth editing instructions and truth image data.

6. The method according to claim 5, characterized in that, The step of training an initial unified multimodal understanding and generation model based on sample data from the training set includes: randomly sampling sample data from the first training set and the second training set; inputting the sampled sample data into the initial unified multimodal understanding and generation model for image generation processing, and outputting prediction information; determining the prediction loss based on the prediction information and the annotation information of the sample data, and tuning the parameters of the initial unified multimodal understanding and generation model based on the prediction loss.

7. The method according to claim 6, characterized in that, When the sample data comes from the first training set, the prediction information includes predicted visual inspection instructions and predicted image data. Determining the prediction loss based on the prediction information and the annotation information of the sample data includes: determining a first loss based on the predicted visual inspection instructions and the ground truth visual inspection instructions corresponding to the sample data; determining a second loss based on the predicted image data and the ground truth image data corresponding to the sample data; and determining the prediction loss based on the first loss and the second loss.

8. The method according to claim 6, characterized in that, When the sample data comes from the second training set, the prediction information includes prediction editing instructions and prediction image data. The step of determining the prediction loss based on the prediction information and the annotation information of the sample data includes: determining a third loss based on the prediction editing instructions and the ground truth editing instructions corresponding to the sample data; determining a fourth loss based on the prediction image data and the ground truth image data corresponding to the sample data; and determining the prediction loss based on the third loss and the fourth loss.

9. The method according to claim 4, characterized in that, The reinforcement learning of the first unified multimodal understanding and generation model based on the sample data in the training set includes: inputting sample text prompt data and sample reference image data from the sample data into the first unified multimodal understanding and generation model for image generation processing, and outputting predicted visual inspection instructions and predicted image data; determining a target reward value based on the predicted visual inspection instructions and the predicted image data, as well as the sample text prompt data and sample reference image data; using the target reward value as the reward signal for reinforcement learning, and optimizing the first unified multimodal understanding and generation model based on the reward signal.

10. The method according to claim 9, characterized in that, The step of determining a target reward value based on the predicted visual inspection instruction, the predicted image data, the sample text prompt data, and the sample reference image data includes: determining a first reward value based on the predicted image data and the sample text prompt data, wherein the first reward value is used to characterize the textual semantic consistency between the predicted image data and the sample text prompt data; determining a second reward value based on the predicted image data, the predicted visual inspection instruction, and the sample reference image data, wherein the second reward value is used to characterize the visual feature consistency between the predicted image data and the sample reference image data; and determining a target reward value based on the first reward value and the second reward value.

11. The method according to claim 10, characterized in that, The step of determining the second reward value based on the predicted image data, the predicted visual inspection instructions, and the sample reference image data includes: for any inspection item in the predicted visual inspection instructions, determining the reward value metric model corresponding to the inspection item according to the visual type indicated by the inspection item, and using the reward value metric model to determine the visual feature consistency between the predicted image data and the sample reference image data to obtain the reward value corresponding to the inspection item; and determining the second reward value based on the reward value corresponding to each inspection item.

12. An image generation apparatus, characterized in that, include: The acquisition unit is configured to acquire text prompt data and reference image data; An image generation processing unit is configured to perform image generation processing on a unified multimodal understanding and generation model by inputting the text prompt data and the reference image data, and output target image data. The unified multimodal understanding and generation model includes a display visual planning module and an adaptive visual correction module. The display visual planning module generates visual inspection instructions based on the text prompt data and the reference image data. These visual inspection instructions guide the adaptive visual correction module to adaptively correct the currently generated image data and ultimately output the target image data.

13. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image generation method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the image generation method as described in any one of claims 1 to 11.

Images