Visual content generation method, apparatus, and electronic device

Abstract

The present disclosure relates to a visual content generation method, device and electronic equipment; the visual content generation method comprises: acquiring original visual content and original prompt text used for generating the original visual content; determining, by a pre-trained multi-modal model, first reverse text according to the original visual content, the first reverse text being used for describing the original visual content; reconstructing the original prompt text according to the first reverse text to obtain reconstructed prompt text; and generating target visual content according to the reconstructed prompt text by a pre-trained diffusion model. The visual content generation method, device and electronic equipment can improve the quality of visual content generation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method, apparatus, and electronic device for generating visual content. Background Technology

[0002] With the development of artificial intelligence technology, it is increasingly being applied to visual content generation. Diffusion models, which generate visual content through progressive denoising, offer significant advantages in terms of detail richness and imaginative appeal, and are widely used in various visual content generation scenarios.

[0003] However, due to various factors, the quality of visual content generated by diffusion models is difficult to guarantee in related technologies. Summary of the Invention

[0004] The purpose of this disclosure is to provide a visual content generation method, apparatus, and electronic device that can improve the quality of visual content generation.

[0005] To achieve the above objectives, in a first aspect, this disclosure provides a visual content generation method, comprising: acquiring original visual content and original prompt text for generating the original visual content; determining a first reverse text based on the original visual content using a pre-trained multimodal model, wherein the first reverse text is used to describe the original visual content; reconstructing the original prompt text based on the first reverse text to obtain reconstructed prompt text; and generating target visual content based on the reconstructed prompt text using a pre-trained diffusion model.

[0006] Optionally, the step of reconstructing the original prompt text based on the first reverse text to obtain the reconstructed prompt text includes: determining the quality of the original visual content based on the original prompt text and the first reverse text using a pre-trained large language model; and reconstructing the original prompt text based on the quality of the original visual content to obtain the reconstructed prompt text.

[0007] Optionally, determining the quality of the original visual content based on the original prompt text and the first reverse text using a pre-trained large language model includes: determining the semantic similarity between the original prompt text and the first reverse text using a pre-trained large language model; determining the matching degree based on the original prompt text and the first reverse text using a pre-trained large language model, wherein the matching degree is used to characterize the matching degree between the original prompt text and the original visual content; and determining the quality of the original visual content based on the semantic similarity and the matching degree using a pre-trained large language model.

[0008] Optionally, the step of reconstructing the original prompt text based on the quality of the original visual content to obtain reconstructed prompt text includes: when the quality of the original visual content does not meet preset quality requirements, determining the text to be added corresponding to the original prompt text and determining the text to be modified included in the original prompt text through a pre-trained large language model, wherein the text to be modified includes redundant text and text unrelated to the original visual content; and reconstructing the original prompt text based on the text to be added and the text to be modified to obtain reconstructed prompt text.

[0009] Optionally, generating target visual content based on the reconstructed prompt text using a pre-trained diffusion model includes: performing iterative noise reduction on the reconstructed prompt text using the pre-trained diffusion model to generate visual content; controlling the iterative noise reduction of the pre-trained diffusion model during the process of generating visual content by iterative noise reduction on the reconstructed prompt text using the pre-trained diffusion model; and determining the visual content generated by the pre-trained diffusion model under the condition of stopping iterative noise reduction as the target visual content in response to the pre-trained diffusion model satisfying the condition of stopping iterative noise reduction.

[0010] Optionally, the iterative denoising control of the pre-trained diffusion model includes: in response to the current iteration number of the pre-trained diffusion model reaching a preset iteration number, evaluating the iterative denoising of the pre-trained diffusion model to obtain evaluation information, wherein the evaluation information is used to characterize whether the iterative denoising of the pre-trained diffusion model is normal, abnormal, or excellent; and performing iterative denoising control on the pre-trained diffusion model based on the evaluation information.

[0011] Optionally, evaluating the iterative denoising of the pre-trained diffusion model to obtain evaluation information includes: acquiring intermediate visual content generated by the pre-trained diffusion model at the current iteration number and historical visual content generated at iteration numbers prior to the current iteration number; determining the quality of the intermediate visual content based on the intermediate visual content, the reconstruction prompt text, and the historical visual content; determining the semantic matching degree between the reconstruction prompt text and the intermediate visual content; determining the potential deviation risk of the intermediate visual content based on the quality of the intermediate visual content and the semantic matching degree; and determining evaluation information based on the quality of the intermediate visual content and the potential deviation risk.

[0012] Optionally, determining the quality of the intermediate visual content based on the intermediate visual content, the reconstructed prompt text, and the historical visual content includes: determining a first semantic similarity between the intermediate visual content and the reconstructed prompt text; determining a second semantic similarity between the historical visual content and the reconstructed prompt text; determining semantic similarity change information based on the first semantic similarity and the second semantic similarity; determining visual similarity between the intermediate visual content and the historical visual content; and determining the quality of the intermediate visual content based on the first semantic similarity, the semantic similarity change information, and the visual similarity.

[0013] Optionally, determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content includes: determining a second reverse text based on the intermediate visual content using a pre-trained multimodal model, wherein the second reverse text is used to describe the intermediate visual content; determining the semantic matching degree between the reconstructed prompt text and the second reverse text; and determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content based on the semantic matching degree between the reconstructed prompt text and the second reverse text.

[0014] Optionally, determining the semantic matching degree between the reconstruction prompt text and the second reverse text includes: determining from the reconstruction prompt text a first text describing entities included in the visual content to be generated, a second text describing attributes of the visual content to be generated, and a third text describing spatial relationships included in the visual content to be generated; determining from the second reverse text a fourth text describing entities included in the intermediate visual content, a fifth text describing attributes of the intermediate visual content, and a sixth text describing spatial relationships included in the intermediate visual content; determining entity semantic matching degree based on the first text and the fourth text; determining attribute semantic matching degree based on the second text and the fifth text; determining spatial relationship semantic matching degree based on the third text and the sixth text; and determining the semantic matching degree between the reconstruction prompt text and the second reverse text based on the entity semantic matching degree, the attribute semantic matching degree, and the spatial relationship semantic matching degree.

[0015] Optionally, the step of performing iterative denoising control on the pre-trained diffusion model based on the evaluation information includes: when the evaluation information indicates that the iterative denoising of the pre-trained diffusion model is normal, controlling the pre-trained diffusion model to continue performing the next iteration of denoising; when the evaluation information indicates that the iterative denoising of the pre-trained diffusion model is excellent, controlling the pre-trained diffusion model to continue performing the next iteration of denoising, and reducing the frequency of iterative denoising control on the pre-trained diffusion model.

[0016] Optionally, the step of performing iterative noise reduction control on the pre-trained diffusion model based on the evaluation information includes: when the evaluation information indicates that the pre-trained diffusion model is abnormal in iterative noise reduction, controlling the pre-trained diffusion model to stop iterative noise reduction based on the reconstructed prompt text; determining a new prompt text; performing iterative noise reduction based on the new prompt text using the pre-trained diffusion model to generate visual content; and performing iterative noise reduction control on the pre-trained diffusion model again during the process of generating visual content through iterative noise reduction based on the new prompt text using the pre-trained diffusion model.

[0017] Optionally, determining the new prompt text includes: determining the new prompt text using a pre-trained large language model, based at least on the evaluation information, the reconstructed prompt text, and the original prompt text.

[0018] Secondly, this disclosure provides a visual content generation apparatus, comprising: an acquisition module for acquiring original visual content and original prompt text for generating the original visual content; a determination module for determining a first reverse text based on the original visual content using a pre-trained multimodal model, wherein the first reverse text describes the original visual content; a reconstruction module for reconstructing the original prompt text based on the original prompt text and the first reverse text to obtain reconstructed prompt text; and a generation module for generating target visual content based on the reconstructed prompt text using a pre-trained diffusion model.

[0019] Thirdly, this disclosure provides an electronic device, including: a memory storing a computer program thereon; and a processor for executing the computer program in the memory to implement the visual content generation method described in the first aspect of this disclosure.

[0020] The above technical solution obtains the original visual content and the original prompt text used to generate the original visual content. A multimodal model determines the first reverse text based on the original visual content. Based on this first reverse text, the original prompt text is reconstructed to obtain the reconstructed prompt text. A pre-trained diffusion model then generates the target visual content based on the reconstructed prompt text. Since the first reverse text describes the actual situation of the original visual content, and the original prompt text describes the required visual content, reconstructing the original prompt text using the first reverse text optimizes the original prompt text and solves the problem of inaccurate original prompt text. Furthermore, the diffusion model's generation of the target visual content based on the reconstructed prompt text addresses the issue of poor quality generated visual content due to inaccurate original prompt text, thereby improving the quality of the visual content generated by the diffusion model.

[0021] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a visual content generation method according to an exemplary embodiment.

[0023] Figure 2 This is a flowchart illustrating an initial image generation stage based on a diffusion model, according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating a text consistency verification phase according to an exemplary embodiment.

[0025] Figure 4 This is a flowchart illustrating a dynamic optimization decision-making stage according to an exemplary embodiment.

[0026] Figure 5 This is a flowchart illustrating the application of a diffusion model in an iterative generation control phase according to an exemplary embodiment.

[0027] Figure 6 This is an iterative evaluation flowchart illustrating an iterative generation control phase according to an exemplary embodiment.

[0028] Figure 7 This is a control flow diagram based on iterative evaluation results in an iterative generation control phase, as illustrated in an exemplary embodiment.

[0029] Figure 8 This is a block diagram illustrating a visual content generation apparatus according to an exemplary embodiment.

[0030] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0031] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0032] With the development of artificial intelligence technology, it is increasingly being applied to visual content generation. Diffusion models, which generate visual content through progressive denoising, offer significant advantages in terms of detail richness and imaginative appeal, and are widely used in various visual content generation scenarios.

[0033] However, due to various factors, the quality of visual content generated by diffusion models is difficult to guarantee in related technologies.

[0034] For example, in related technologies, users need to adjust the prompt text of the input diffusion model multiple times to improve the quality of the generated visual content. This approach not only reduces the efficiency of visual content generation but is also susceptible to user subjectivity, making it difficult to improve the quality of the generated visual content.

[0035] An example of an image or video generation process includes: Input prompts and noise: The user inputs prompt text and noise image information. Progressive denoising generation: The model generates image or video content step by step through multi-step iterative denoising. Output result: The generated image or video content is directly output.

[0036] The shortcomings of this generation method include: Inconsistency between generated content and prompts: The diffusion model lacks a deep understanding and verification mechanism for the prompt content during the generation process, potentially leading to results that deviate from user expectations. Limited generation quality: The potential of the diffusion model has not been fully realized; there is still room for improvement in the richness of detail, logical coherence, and consistency of the generated content. Inadequate user experience: Users need to adjust prompts or regenerate content multiple times, reducing generation efficiency and user experience.

[0037] Based on this, the present disclosure provides a technical solution to obtain original visual content and original prompt text for generating the original visual content, determine a first reverse text based on the original visual content using a multimodal model, reconstruct the original prompt text based on the first reverse text to obtain reconstructed prompt text, and generate target visual content based on the reconstructed prompt text using a pre-trained diffusion model.

[0038] Since the first reverse text can describe the actual situation of the original visual content and the original prompt text can describe the required visual content, the original prompt text can be optimized by reconstructing it using the first reverse text, thus solving the problem of inaccurate original prompt text. Furthermore, the diffusion model generates the target visual content based on the reconstructed prompt text, which can solve the problem of poor quality of the generated visual content caused by the inaccuracy of the original prompt text, thereby improving the quality of the visual content generated by the diffusion model.

[0039] In some embodiments, the quality of visual content may involve the resolution of the visual content, the consistency between the visual content and the prompt text, and the logical rationality of the visual content.

[0040] In some embodiments, the visual content may be an image or a video, without limitation.

[0041] The technical solutions of this disclosure can be applied to various scenarios such as image generation, video production, virtual reality, augmented reality, film and television special effects, and advertising design.

[0042] The type of visual content may differ in different application scenarios. For example, in a landscape-related visual content generation scenario, it might be necessary to generate images or videos of landscape objects such as trees, sky, and lakes. In a traffic-related visual content generation scenario, it might be necessary to generate images or videos of traffic objects such as pedestrians, vehicles, traffic lights, and roads.

[0043] Figure 1 This is a flowchart illustrating a visual content generation method according to an exemplary embodiment, the visual content generation method comprising the following steps: Step S11: Obtain the original visual content and the original prompt text used to generate the original visual content.

[0044] Step S12: Using a pre-trained multimodal model, determine the first reverse text based on the original visual content. The first reverse text is used to describe the original visual content.

[0045] Step S13: Reconstruct the original prompt text based on the first reverse text to obtain the reconstructed prompt text.

[0046] Step S14: Using a pre-trained diffusion model, generate the target visual content based on the reconstructed prompt text.

[0047] In step S11, the original visual content can be visual content generated based on the original prompt text.

[0048] In some embodiments, the original prompt text can be input into a pre-trained diffusion model, and the pre-trained diffusion model can iteratively reduce noise based on the original prompt text to output the original visual content.

[0049] In some embodiments, the input to the diffusion model may include not only the original prompt text, but also information such as the size of the visual content, for example, image size, video size, etc.

[0050] In some embodiments, the original prompt text may include various texts describing the visual content to be generated.

[0051] For example, if the visual content to be generated includes multiple entities, the original prompt text may include prompt text describing these multiple entities; if the visual content to be generated has corresponding attributes, the original prompt text may include prompt text describing the attributes; if the visual content to be generated includes spatial relationships, the original prompt text may include prompt text describing the spatial relationships.

[0052] In some embodiments, the application of diffusion models is a mature technology in the art. Therefore, how the diffusion model performs iterative denoising based on relevant information of the prompt text and visual content (i.e., the specific iterative denoising process) to generate visual content is not described in detail in this disclosure. Correspondingly, the training of the diffusion model is also a mature technology in the art and will not be described in detail here.

[0053] In some embodiments, if no corresponding iteration control is performed on the diffusion model, the diffusion model can stop iterating after reaching the corresponding number of iterations, and the generated visual content at this time is the visual content output by the diffusion model.

[0054] A multimodal model is an artificial intelligence model capable of processing multiple types of data, such as text, images, audio, and video. The goal of such a model is to provide more comprehensive and accurate information by combining different types of data. For example, a multimodal model can analyze information not only in text data but also in image and audio data. In this way, multimodal models can provide richer and deeper understandings and insights.

[0055] Therefore, multimodal models can be used to analyze the information included in the original visual content to obtain the corresponding text information describing the original visual content, which is the first reverse text.

[0056] For example, if the original visual content includes multiple entities, the first reverse text may include text describing these multiple entities; if the original visual content has corresponding attributes, the first reverse text may include text describing the attributes; if the original visual content includes spatial relationships, the first reverse text may include text describing the spatial relationships.

[0057] Regarding multimodal models, these could include: Transformer models (deep learning models based on self-attention mechanisms), ResNet (Residual Network, deep convolutional network) models, etc.

[0058] The application and training of multimodal models are mature technologies in this field, therefore, the application and training methods of multimodal models will not be described in detail here.

[0059] As an optional implementation, step S13 includes: determining the quality of the original visual content based on the original prompt text and the first reverse text using a pre-trained large language model; and reconstructing the original prompt text based on the quality of the original visual content to obtain the reconstructed prompt text.

[0060] In this implementation, a large language model is used to combine the original prompt text and the first reverse text to first assess the quality of the original visual content, and then the original prompt text is reconstructed based on the quality assessment results to obtain the reconstructed prompt text.

[0061] In some embodiments, the quality of the original visual content is determined based on the original prompt text and the first reverse text using a pre-trained large language model, including: determining the semantic similarity between the original prompt text and the first reverse text using the pre-trained large language model; determining the matching degree based on the original prompt text and the first reverse text using the pre-trained large language model, wherein the matching degree is used to characterize the matching degree between the original prompt text and the original visual content; and determining the quality of the original visual content based on the semantic similarity and the matching degree using the pre-trained large language model.

[0062] In this implementation, the large language model evaluates the quality of the original visual content from two dimensions: the first dimension is the semantic similarity between the original prompt text and the first reverse text, and the second dimension is the matching degree between the original prompt text and the original visual content.

[0063] The semantic similarity between the original prompt text and the first reverse text, representing the similarity between the texts, can be determined using a text similarity algorithm.

[0064] The degree of matching between the original prompt text and the original visual content can be determined by determining the semantic matching degree between the original prompt text and the first reverse text. That is, the semantic matching degree between the original prompt text and the first reverse text represents the degree of matching between the original prompt text and the original visual content.

[0065] The matching degree between the original prompt text and the first reverse text can be determined by analyzing the matching degree between the text describing the entity, the text describing the attribute, and the text describing the spatial relationship, and then performing a quantitative evaluation. For details on the specific analysis method for the matching degree, please refer to the description in the subsequent embodiments.

[0066] In some embodiments, semantic similarity and matching degree can be combined to determine the quality of the original visual content.

[0067] In some embodiments, semantic similarity and matching degree can be directly used as information to characterize the quality of the original visual content.

[0068] In some embodiments, semantic similarity and matching degree can be weighted and quantified into a specific quality score, which is not limited here.

[0069] In some embodiments, reconstructing the original prompt text based on the quality of the original visual content to obtain reconstructed prompt text may include: reconstructing the original prompt text to obtain reconstructed prompt text when the quality of the original visual content does not meet preset quality requirements.

[0070] In some embodiments, taking the quality of the original visual content as including both semantic similarity and matching degree as an example, the preset quality requirement can be that the semantic similarity is greater than a preset semantic similarity, and the matching degree is greater than a preset matching degree. The preset semantic similarity and preset matching degree can be set according to different application scenarios, and their values ​​are not limited here.

[0071] In some embodiments, the quality of the original visual content is taken as an example, which is the quality score obtained by semantic similarity and matching quantification. The preset quality requirement can be that the quality score is higher than the preset quality score. The preset quality score can be set according to different application scenarios, and no limit is imposed on the value here.

[0072] In some embodiments, if the quality of the original visual content meets the preset quality requirements, the original visual content can be directly used as the target visual content without performing the text reconstruction enhancement operation.

[0073] Taking the generation of original visual content through a diffusion model as an example, the diffusion model generates original visual content through iterative noise reduction. The original visual content can be the visual content generated in any iteration. Then, if the quality of the original visual content meets the preset quality requirements (quality meets the standard), the iteration of the diffusion model can be stopped, and the original visual content can be used as the final target visual content.

[0074] In some embodiments, reconstructing the original prompt text based on the quality of the original visual content to obtain reconstructed prompt text may include: if the quality of the original visual content does not meet preset quality requirements, determining the text to be added corresponding to the original prompt text and determining the text to be modified included in the original prompt text through a pre-trained large language model, wherein the text to be modified includes redundant text and text unrelated to the original visual content; and reconstructing the original prompt text based on the text to be added and the text to be modified to obtain reconstructed prompt text.

[0075] In this implementation, a pre-trained large language model is used to perform semantic deviation diagnosis on the original prompt text, thereby identifying the text that needs to be modified (i.e., the text to be modified) and the text that needs to be added (i.e., the text to be added). Then, the text is reconstructed by combining the text to be modified and the text to be added.

[0076] In some embodiments, the text that needs to be modified may include redundant text and text that is not relevant to the original visual content.

[0077] For example, redundant text can be logically incorrect or redundant. For redundant text, one possible modification method is to correct the redundant logical statements.

[0078] For example, text unrelated to the original visual content can be text not mentioned in the first reverse text; this type of text can also be called contradictory elements. For this type of text, a possible modification is reinforcement. Specifically, since this type of text is unrelated to the original visual content and is not reflected in it, it needs to be reinforced so that the diffusion model can notice its influence on the generated visual content. For example, this could be modified into more detailed descriptive text, making it more precise and achieving a reinforcement effect.

[0079] In some embodiments, the text to be added may correspond to key content missing from the original visual content. For example, if the original visual content does not include a certain entity that is a key entity for the visual content, then the descriptive text of that key entity can be used as the text to be added.

[0080] In some embodiments, the original prompt text is reconstructed based on the text to be added and the text to be modified to obtain the reconstructed prompt text, which can also be achieved by a pre-trained large language model.

[0081] Therefore, by identifying missing, redundant, and contradictory elements, these elements are optimized in a targeted manner, including supplementing key descriptions, correcting redundant logical expressions, and strengthening contradictory elements, resulting in reconstructed prompt text. Compared to the original prompt text, this reconstructed prompt text can be used to generate higher-quality visual content.

[0082] In step S14, iterative noise reduction of the diffusion model can be enabled based on the reconstructed prompt text to generate visual content.

[0083] It is understood that the information input into the diffusion model includes not only the reconstructed prompt text, but also information related to the visual content, as can be seen in the aforementioned embodiments.

[0084] As described in the foregoing embodiments, the related technologies lack a deep understanding and verification mechanism for the prompt content, which may cause the generated results to deviate from the user's expectations. In order to solve this problem, the embodiments of this disclosure also provide an iterative control mechanism for the diffusion model. By constructing a closed-loop verification architecture and a dynamic intervention mechanism, the core problems such as content drift, missing / redundant key elements, and logical contradictions during the generation process are solved.

[0085] Therefore, as an optional implementation, step S14 includes: generating visual content by iteratively denoising based on the reconstructed prompt text using a pre-trained diffusion model; controlling the iterative denoising of the pre-trained diffusion model during the process of generating visual content by iteratively denoising based on the reconstructed prompt text using the pre-trained diffusion model; and determining the visual content generated by the pre-trained diffusion model under the condition of stopping iterative denoising as the target visual content in response to the pre-trained diffusion model satisfying the condition of stopping iterative denoising.

[0086] In some embodiments, visual content is generated by iteratively denoising based on the reconstructed prompt text using a pre-trained diffusion model. This can be understood as inputting relevant information of the reconstructed prompt text and the visual content into the diffusion model so that the diffusion model can begin iteratively denoising to generate visual content.

[0087] In some embodiments where the original visual content is also generated by the diffusion model, the relevant information of the visual content has already been input into the diffusion model and does not need to be input again.

[0088] Furthermore, during the process of generating visual content through iterative noise reduction using a pre-trained diffusion model based on reconstructed prompt text, iterative noise reduction control is applied to the pre-trained diffusion model.

[0089] In some embodiments, iterative denoising control of a pre-trained diffusion model includes: in response to the current iteration number of the pre-trained diffusion model reaching a preset iteration number, evaluating the iterative denoising of the pre-trained diffusion model to obtain evaluation information, the evaluation information being used to characterize whether the iterative denoising of the pre-trained diffusion model is normal, abnormal, or excellent; and performing iterative denoising control on the pre-trained diffusion model based on the evaluation information.

[0090] In this implementation, the iterative noise reduction performance is evaluated for a specified number of iterations, and corresponding iterative noise reduction control is performed based on the evaluation results.

[0091] In some embodiments, the preset number of iterations can be a critical number of iterations. For example, the preset number of iterations can be 30%, 50%, 70%, etc., of the total number of iterations.

[0092] In some embodiments, the total number of iterations of the diffusion model is usually preset and does not change.

[0093] Therefore, the iterative noise reduction performance of the diffusion model can be evaluated at key iteration steps to achieve multi-step thinking and verification.

[0094] In this embodiment of the disclosure, the evaluation of iterative noise reduction may involve process output quality assessment and potential deviation risk prediction. Process output quality assessment can be understood as evaluating the quality of the currently generated visual content, and potential deviation risk prediction can be understood as predicting the risk of deviating from the visual content generation requirements.

[0095] Therefore, as an optional implementation, the iterative denoising of the pre-trained diffusion model is evaluated to obtain evaluation information, including: acquiring the intermediate visual content generated by the pre-trained diffusion model at the current iteration number and the historical visual content generated at iteration numbers prior to the current iteration number; determining the quality of the intermediate visual content based on the intermediate visual content, the reconstructed prompt text, and the historical visual content; determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content; determining the potential bias risk of the intermediate visual content based on the quality of the intermediate visual content and the semantic matching degree; and determining evaluation information based on the quality of the intermediate visual content and the potential bias risk.

[0096] In some embodiments, the number of iterations prior to the current iteration number may refer to the one and two iterations before the current iteration number.

[0097] In some embodiments, determining the quality of intermediate visual content based on intermediate visual content, reconstructed prompt text, and historical visual content may include: determining a first semantic similarity between the intermediate visual content and the reconstructed prompt text; determining a second semantic similarity between the historical visual content and the reconstructed prompt text; determining semantic similarity change information based on the first semantic similarity and the second semantic similarity; determining visual similarity between the intermediate visual content and the historical visual content; and determining the quality of the intermediate visual content based on the first semantic similarity, semantic similarity change information, and visual similarity.

[0098] In some embodiments, semantic similarity can be determined using the CLIP (Contrastive Language-Image Pre-training, a multimodal pre-trained neural network model).

[0099] It is understandable that the CLIP model can map images and text to the same semantic space through contrastive learning, thereby achieving cross-modal semantic alignment and determining the semantic similarity between images and text.

[0100] In some embodiments, the visual similarity between intermediate visual content and historical visual content can be determined using the LPIPS (Learned Perceptual Image PatchSimilarity, a deep learning-based image similarity metric) model.

[0101] The LPIPS model extracts high-level features from images using deep convolutional neural networks and calculates the distances between these features to evaluate the perceptual similarity of images.

[0102] It is understandable that when the visual content is an image, the two models described above can be processed directly based on the image. When the visual content is video, the video can be divided into multiple video image frames, and the two models described above can be processed based on these video image frames.

[0103] Furthermore, the first semantic similarity, semantic similarity change information, and visual similarity can be weighted and summed to obtain a quantitative result representing the quality of the intermediate visual content.

[0104] As an example, the quality of intermediate visual content can be represented as:

[0105]

[0106] The meanings of the parameters in the above calculation formula are as follows: t: The current iteration step, also known as the current iteration count, can represent the t-th iteration.

[0107] I t : The visual content generated in the t-th iteration, i.e., the intermediate visual content.

[0108] P opt : Reconstruct the prompt text.

[0109] k: The sliding window size, used to determine how many historical visuals to take before the current iteration. For example, if the historical visuals involve two iterations before the current iteration, then k could be 2.

[0110] i: Different values ​​of i correspond to different historical visual content. For example, if i is 1, the corresponding historical visual content is the visual content generated in the previous iteration.

[0111] S CLIP The CLIP model calculates the semantic similarity (cosine similarity) between the cue text and the visual content to evaluate the degree of semantic alignment.

[0112] D LIPPS The LPIPS model calculates the perceptual similarity score between visual content, assesses the perceptual similarity between the current iteration image and the previous step image, and is used to determine the visual similarity between intermediate visual content and historical visual content.

[0113] S CLIPSemantic similarity change information can characterize the changes in the first and second semantic similarities. This information can be used as a trend correction term to calculate the changing trend of CLIP scores using a sliding window, identifying continuous improvements in semantic alignment.

[0114] α1: The weight corresponding to the first semantic similarity, which can be a fixed value to maintain the stability of semantic evaluation. For example, the fixed value can be 0.7.

[0115] β1: The weight corresponding to visual similarity, which can be expressed as: ,in, The time constant can be 10, and this weight can be gradually increased with the number of iterations to achieve detailed improvement evaluation.

[0116] γ1: The weight corresponding to the semantic similarity change information. It can be represented as: This means that when quality continues to improve, this weight is a reward weight; when quality deteriorates, this weight is a penalty weight.

[0117] In some embodiments, determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content may include: determining a second reverse text based on the intermediate visual content using a pre-trained multimodal model, the second reverse text being used to describe the intermediate visual content; determining the semantic matching degree between the reconstructed prompt text and the second reverse text; and determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content based on the semantic matching degree between the reconstructed prompt text and the second reverse text.

[0118] In this implementation, a second reverse text for describing the intermediate visual content is first determined based on the intermediate visual content. The method for determining the second reverse text can refer to the aforementioned embodiment of the first reverse text, and will not be repeated here.

[0119] In some embodiments, determining the semantic matching degree between the reconstruction prompt text and the second reverse text can be implemented by a pre-trained large language model. This determination process may include: determining from the reconstruction prompt text a first text describing entities included in the visual content to be generated, a second text describing attributes of the visual content to be generated, and a third text describing spatial relationships included in the visual content to be generated; determining from the second reverse text a fourth text describing entities included in the intermediate visual content, a fifth text describing attributes of the intermediate visual content, and a sixth text describing spatial relationships included in the intermediate visual content; determining the entity semantic matching degree based on the first and fourth texts; determining the attribute semantic matching degree based on the second and fifth texts; determining the spatial relationship semantic matching degree based on the third and sixth texts; and determining the semantic matching degree between the reconstruction prompt text and the second reverse text based on the entity semantic matching degree, attribute semantic matching degree, and spatial relationship semantic matching degree.

[0120] The visual content to be generated can be understood as the visual content generated according to requirements.

[0121] It is understandable that the method for determining the semantic matching degree between the original prompt text and the first reverse text mentioned above can refer to the method for determining the semantic matching degree between the reconstructed prompt text and the second reverse text here.

[0122] In some embodiments, entity semantic matching degree, attribute semantic matching degree, and spatial relationship semantic matching degree can be determined by calculating the similarity between the corresponding texts.

[0123] As an example, semantic matching degree can be represented as:

[0124]

[0125]

[0126]

[0127] The relevant parameters in the above calculation formula are explained as follows: S match : Reconstruct the semantic matching degree between the prompt text and the second reverse text.

[0128] S E Entity semantic matching degree.

[0129] S A Attribute semantic matching degree.

[0130] S R Semantic matching degree of control relationships.

[0131] n: The total number of texts for which semantic matching needs to be calculated.

[0132] sim: represents a similarity determination algorithm.

[0133] : Reconstruct the i-th text in the prompt text that describes the entity, i.e. the i-th first text.

[0134] The i-th text in the second reverse text used to describe the entity is the i-th third text.

[0135] : Reconstruct the i-th text in the prompt text that describes the attribute, i.e., the i-th second text.

[0136] The i-th text in the second reverse text that describes the attribute is the i-th fourth text.

[0137] : The i-th text in the reconstructed prompt text that describes spatial relationships, i.e., the i-th third text.

[0138] The i-th text in the second reverse text used to describe spatial relationships is the i-th sixth text.

[0139] The weight corresponding to the semantic matching degree of the entity can be configured with different values ​​depending on the application scenario.

[0140] The weight corresponding to the semantic matching degree of the attribute can be configured with different values ​​depending on the application scenario.

[0141] The weight corresponding to the semantic matching degree of spatial relationship can be configured with different values ​​according to different application scenarios.

[0142] Furthermore, potential bias risks can be determined based on the quality and semantic matching degree of the intermediate visual content.

[0143] As an example, the potential risk of deviation can be expressed as:

[0144] Among them, R risk This represents a potential risk of deviation. This represents the contribution weight of semantic matching degree to potential risk, and its value can range from 0.6 to 0.8. For example, it can be 0.7.

[0145] In some embodiments, it can be determined whether the iteration is normal based on the quality of the intermediate visual content and the potential risk of deviation.

[0146] In some embodiments, a minimum quality assessment value Q is pre-configured according to the requirements of the application scenario. min and maximum quality assessment value Q max and the lowest potential deviation risk R min .

[0147] As an example, in Q score ≥Q max And R risk <R min Under these circumstances, it is determined that the pre-trained diffusion model has excellent iterative noise reduction performance.

[0148] As an example, in Q min ≤Q score <Q max And R risk ≥R min In this case, it is determined that the pre-trained diffusion model's iterative noise reduction is normal.

[0149] As an example, in Q score ≤Q min And R risk ≥R min In this case, it is determined that the pre-trained diffusion model has an abnormal iterative denoising.

[0150] In some embodiments, the evaluation information may include, in addition to characterizing whether the pre-trained diffusion model's iterative noise reduction is normal, abnormal, or excellent, the quality of the intermediate visual content and the potential risk of bias.

[0151] In some embodiments, iterative denoising control of the pre-trained diffusion model based on evaluation information may include: if the evaluation information indicates that the iterative denoising of the pre-trained diffusion model is normal, controlling the pre-trained diffusion model to continue to perform the next iteration of denoising; if the evaluation information indicates that the iterative denoising of the pre-trained diffusion model is excellent, controlling the pre-trained diffusion model to continue to perform the next iteration of denoising, and reducing the frequency of iterative denoising control of the pre-trained diffusion model.

[0152] If the pre-trained diffusion model performs normally in the iterative denoising, then the pre-trained diffusion model can be controlled to continue the next iteration of denoising, and the subsequent iterative denoising process of the pre-trained diffusion model can be controlled.

[0153] If the pre-trained diffusion model performs well in iterative denoising, it can be controlled to continue performing the next iteration of denoising, and the subsequent iterative denoising process of the pre-trained diffusion model can be further controlled. Furthermore, the frequency of iterative denoising control of the pre-trained diffusion model can be reduced.

[0154] For example, as described in the foregoing embodiments, the evaluation of iterative denoising is only performed at critical iteration steps / numbers. Therefore, if the pre-trained diffusion model has excellent iterative denoising performance, the evaluation of iterative denoising at the next critical iteration step can be skipped.

[0155] Specifically, assuming the critical iteration steps are 30%, 50%, and 70%, if the evaluation result obtained at 30% of the iteration steps is excellent, then at 50%, iterative noise reduction evaluation is not required.

[0156] The iterative noise reduction control method described above can be called iterative control based on the rejection sampling mechanism.

[0157] In some embodiments, when an abnormality occurs in the iterative denoising of the pre-trained diffusion model for evaluating information representation, the pre-trained diffusion model is controlled to stop iterative denoising based on the reconstructed prompt text; a new prompt text is determined; the pre-trained diffusion model is used to iteratively denoise based on the new prompt text to generate visual content; during the process of generating visual content through iterative denoising based on the new prompt text using the pre-trained diffusion model, the pre-trained diffusion model is again subjected to iterative denoising control.

[0158] In the event of an anomaly during iterative noise reduction, the current iteration needs to be stopped. Furthermore, a new prompt text needs to be determined to initiate a new round of iterative noise reduction based on the new prompt text, generating visual content.

[0159] In some embodiments, determining new prompt text includes: determining new prompt text using a pre-trained large language model, based at least on evaluation information, reconstructed prompt text, and original prompt text.

[0160] In this implementation, a pre-trained large language model can be used to identify abnormal features by combining evaluation information, reconstructed prompt text, and original prompt text. This allows for reverse deduction of the root cause of the problem, and based on this root cause analysis, a new prompt text with compensatory constraints can be generated. This enables the initiation of a new round of diffusion generation.

[0161] It is understandable that the target visual content can be obtained when the pre-trained diffusion model meets the stopping iteration noise reduction condition.

[0162] The conditions for stopping iterative noise reduction may include, but are not limited to: the user actively exiting the visual content generation process; the quality of the generated visual content meeting the standards (refer to the aforementioned implementation method for quality assessment); and reaching the preset number of iterative noise reduction iterations.

[0163] It is understandable that the above-mentioned stopping iterative denoising condition applies to any round of diffusion model iterative denoising to generate visual content.

[0164] In some embodiments, the diffusion model can output target visual content that meets quality standards.

[0165] In some embodiments, the pre-trained large language model can also output a process optimization report, which can record the process of continuously optimizing the reconstructed prompt text, or record the source information that leads to the anomaly, etc., without limitation.

[0166] The technical solution of this disclosure embodiment may involve an application process in two stages: the first stage is the initial generation stage, and the second stage is the multi-step thinking and verification stage. The multi-step thinking and verification stage may include: a text consistency verification stage, a dynamic optimization decision stage, and an iterative generation control stage.

[0167] The following section uses images as an example of visual content to introduce a specific application process.

[0168] Figure 2 This is a flowchart illustrating an initial image generation stage based on a diffusion model, according to an exemplary embodiment, such as... Figure 2 As shown, in the initial image generation stage, the user inputs prompt text and image size information, and the initial image is generated through iterative noise reduction based on the diffusion model with default parameters.

[0169] Figure 3 This is a flowchart illustrating a text consistency verification phase according to an exemplary embodiment, such as... Figure 3 As shown, the text consistency verification stage requires the application of the prompt text and the generated initial image from the initial image generation stage.

[0170] In this text consistency verification process, reverse text is extracted using a multimodal model to obtain reverse text, and then a large language model is used to perform semantic consistency discrimination and key information matching to obtain the image quality judgment result.

[0171] Figure 4 This is a flowchart illustrating a dynamic optimization decision-making stage according to an exemplary embodiment, such as... Figure 4 As shown, the dynamic optimization decision-making stage requires the application of the image quality judgment results obtained in the text consistency verification stage.

[0172] In this dynamic optimization decision-making process, if the quality meets the standards, the generated image can be directly output.

[0173] If the quality is not up to standard, semantic deviation diagnosis and reconstruction of prompt text are performed using a large language model to obtain the reconstructed prompt text.

[0174] Figure 5 This is an exemplary embodiment illustrating the application flowchart of a diffusion model in an iterative generation control phase, in which the refactoring prompt text determined in the dynamic optimization decision phase needs to be applied.

[0175] Based on the reconstructed prompt text, a new round of diffusion generation can be performed. Once the maximum number of iterations is reached, the generated image can be output.

[0176] If the maximum number of iteration steps has not been reached, determine whether it is a critical iteration step. If so, further determine whether to skip this sampling step (based on the iteration evaluation result of the previous critical iteration step). If the sampling step is skipped, continue iterating. If the sampling step is not skipped, the iteration evaluation process needs to be executed.

[0177] Figure 6 This is a flowchart illustrating an iterative evaluation process in an iterative generation control phase according to an exemplary embodiment. Figure 6 In this study, for images requiring iterative evaluation, the CLIP model is used to calculate the semantic matching degree between the prompt text and the generated image; and the LPIPS model is used to calculate the perceptual similarity between the generated images. The image quality assessment result is determined by combining the semantic matching degree and the perceptual similarity.

[0178] Furthermore, reverse text extraction is performed using a multimodal model, and then a large language model is used to extract key information and evaluate semantic matching based on the reverse text, thereby obtaining the semantic matching degree between the reverse text and the reconstructed prompt text.

[0179] Furthermore, by combining image quality assessment results and semantic matching degree, potential bias risks can be identified. Based on these potential bias risks, iterative evaluations can be performed.

[0180] Figure 7 This is a control flow diagram based on iterative evaluation results in an iterative generation control phase, as illustrated in an exemplary embodiment. Figure 7 As shown, different control methods are adopted based on three different iterative evaluation results.

[0181] If the evaluation is normal, the iteration can continue, that is, it can continue to be executed as follows. Figure 5 The process is shown below.

[0182] In cases of excellent evaluation, not only can iteration continue, but it can also be decided to skip the next sampling evaluation, or it can correspond to... Figure 5 The process is shown below.

[0183] If an anomaly is detected, the current iteration is terminated. Furthermore, if the maximum number of iterations has not been reached, a large language model can be used to perform reverse problem tracing and generate compensating constraint prompt text to obtain new prompt text, and then the process can proceed as follows: Figure 5 The process is shown below.

[0184] It is understood that an iteration includes multiple iteration steps. Therefore, the iteration round can be understood as the number of times visual content is generated for the diffusion model, and the iteration step can be understood as the number of iteration noise reduction steps in each iteration. The iteration number in the aforementioned embodiment has the same meaning as the iteration step.

[0185] When the maximum number of iterations is reached, a process optimization report can be generated.

[0186] Combination Figures 2-7 As can be seen from the flowchart, the technical solution of this disclosure addresses the problems in related technologies such as the cumulative semantic deviation between generated content and prompt text, the uncontrollable generation process, and the lack of systematic quality optimization. It proposes a visual content generation method based on multi-step thinking and verification. By constructing a closed-loop verification architecture and a dynamic intervention mechanism, it solves core problems such as content drift, missing / redundant key elements, and logical contradictions during the generation process.

[0187] This technical solution involves the following three-level optimization system: Multimodal comparison and verification layer: By combining reverse text extraction with large language model analysis, a semantic consistency quantitative evaluation system is established to accurately locate the deviation between the generated content and the input prompt.

[0188] Dynamic decision control layer: The generation process is controlled by a rejection sampling mechanism. At key iteration steps (such as 30%, 50%, 70%), process quality assessment and risk prediction are performed, and the generation path is dynamically adjusted.

[0189] Semantic Enhancement Reconstruction Layer: Based on the bias diagnosis and compensation constraint generation of the large language model, iteratively optimizes the semantic expression of the prompt text to ensure logical consistency in multiple rounds of generation.

[0190] Thus, a recursive reinforcement closed loop of "generation-verification-dynamic decision-semantic reconstruction" is formed, and the system stability is ensured by setting the maximum number of rounds and intelligent termination conditions.

[0191] Furthermore, the technical effects that can be achieved may include: Process controllability: The rejection sampling mechanism allows for real-time intervention during the diffusion generation process, solving the lag problem of traditional "post-generation verification".

[0192] Semantic bias correction capability: Through a two-stage diagnostic-optimization process driven by a large language model, semantic drift in multiple iterations is effectively eliminated.

[0193] Generation stability: A three-level evaluation strategy (excellent / normal / abnormal) for key iteration steps enables risk classification and control of the generation process, avoiding ineffective iterations.

[0194] Resource optimization: When the prediction evaluation is excellent, subsequent sampling evaluation is skipped, which significantly improves the generation efficiency while ensuring quality.

[0195] Enhanced interpretability: Output process optimization report, providing semantic deviation correction trajectory and quality iteration path to help users understand the generation logic.

[0196] Figure 8 This is a block diagram illustrating a visual content generation apparatus 800 according to an exemplary embodiment, such as... Figure 8 As shown, the device includes: The acquisition module 801 is used to acquire the original visual content and the original prompt text used to generate the original visual content.

[0197] The determination module 802 is used to determine a first reverse text based on the original visual content using a pre-trained multimodal model. The first reverse text is used to describe the original visual content.

[0198] The reconstruction module 803 is used to reconstruct the original prompt text based on the original prompt text and the first reverse text to obtain the reconstructed prompt text.

[0199] The generation module 804 is used to generate target visual content based on the reconstructed prompt text using a pre-trained diffusion model.

[0200] Optionally, the reconstruction module 803 is further configured to: determine the quality of the original visual content based on the original prompt text and the first reverse text using a pre-trained large language model; and reconstruct the original prompt text based on the quality of the original visual content to obtain a reconstructed prompt text.

[0201] Optionally, the reconstruction module 803 is further configured to: determine the semantic similarity between the original prompt text and the first reverse text using a pre-trained large language model; determine the matching degree based on the original prompt text and the first reverse text using the pre-trained large language model, wherein the matching degree is used to characterize the matching degree between the original prompt text and the original visual content; and determine the quality of the original visual content based on the semantic similarity and the matching degree using the pre-trained large language model.

[0202] Optionally, the reconstruction module 803 is further configured to: when the quality of the original visual content does not meet the preset quality requirements, determine the text to be added corresponding to the original prompt text and determine the text to be modified included in the original prompt text through a pre-trained large language model, wherein the text to be modified includes redundant text and text unrelated to the original visual content; and reconstruct the original prompt text according to the text to be added and the text to be modified to obtain the reconstructed prompt text.

[0203] Optionally, the generation module 804 is further configured to: generate visual content by iteratively denoising the reconstructed prompt text using the pre-trained diffusion model; control the iterative denoising of the pre-trained diffusion model during the process of generating visual content by iterative denoising the reconstructed prompt text using the pre-trained diffusion model; and determine the visual content generated by the pre-trained diffusion model under the condition of stopping iterative denoising as the target visual content in response to the pre-trained diffusion model satisfying the condition of stopping iterative denoising.

[0204] Optionally, the generation module 804 is further configured to: in response to the current iteration number of the pre-trained diffusion model reaching a preset iteration number, evaluate the iterative denoising of the pre-trained diffusion model to obtain evaluation information, wherein the evaluation information is used to characterize whether the iterative denoising of the pre-trained diffusion model is normal, abnormal, or excellent; and perform iterative denoising control on the pre-trained diffusion model based on the evaluation information.

[0205] Optionally, the generation module 804 is further configured to: evaluate the iterative denoising of the pre-trained diffusion model to obtain evaluation information, including: acquiring intermediate visual content generated by the pre-trained diffusion model at the current iteration number and historical visual content generated at iteration numbers prior to the current iteration number; determining the quality of the intermediate visual content based on the intermediate visual content, the reconstruction prompt text, and the historical visual content; determining the semantic matching degree between the reconstruction prompt text and the intermediate visual content; determining the potential deviation risk of the intermediate visual content based on the quality of the intermediate visual content and the semantic matching degree; and determining evaluation information based on the quality of the intermediate visual content and the potential deviation risk.

[0206] Optionally, the generation module 804 is further configured to: determine the quality of the intermediate visual content based on the intermediate visual content, the reconstructed prompt text, and the historical visual content, including: determining a first semantic similarity between the intermediate visual content and the reconstructed prompt text; determining a second semantic similarity between the historical visual content and the reconstructed prompt text; determining semantic similarity change information based on the first semantic similarity and the second semantic similarity; determining visual similarity between the intermediate visual content and the historical visual content; and determining the quality of the intermediate visual content based on the first semantic similarity, the semantic similarity change information, and the visual similarity.

[0207] Optionally, the generation module 804 is further configured to: determine a second reverse text based on the intermediate visual content using a pre-trained multimodal model, wherein the second reverse text is used to describe the intermediate visual content; determine the semantic matching degree between the reconstructed prompt text and the second reverse text; and determine the semantic matching degree between the reconstructed prompt text and the intermediate visual content based on the semantic matching degree between the reconstructed prompt text and the second reverse text.

[0208] Optionally, the generation module 804 is further configured to: determine from the reconstruction prompt text a first text describing the entities included in the visual content to be generated, a second text describing the attributes of the visual content to be generated, and a third text describing the spatial relationships included in the visual content to be generated; determine from the second reverse text a fourth text describing the entities included in the intermediate visual content, a fifth text describing the attributes of the intermediate visual content, and a sixth text describing the spatial relationships included in the intermediate visual content; determine the entity semantic matching degree based on the first text and the fourth text; determine the attribute semantic matching degree based on the second text and the fifth text; determine the spatial relationship semantic matching degree based on the third text and the sixth text; and determine the semantic matching degree between the reconstruction prompt text and the second reverse text based on the entity semantic matching degree, the attribute semantic matching degree, and the spatial relationship semantic matching degree.

[0209] Optionally, the generation module 804 is further configured to: control the pre-trained diffusion model to continue the next iteration of denoising when the evaluation information indicates that the pre-trained diffusion model is performing normally in iterative denoising; and control the pre-trained diffusion model to continue the next iteration of denoising when the evaluation information indicates that the pre-trained diffusion model is performing excellently in iterative denoising, and reduce the frequency of iterative denoising control of the pre-trained diffusion model.

[0210] Optionally, the generation module 804 is further configured to: control the pre-trained diffusion model to stop iterative denoising based on the reconstructed prompt text when the evaluation information indicates that the pre-trained diffusion model is abnormal in iterative denoising; determine a new prompt text; generate visual content by iterative denoising based on the new prompt text using the pre-trained diffusion model; and control the pre-trained diffusion model to iteratively denoise again during the process of generating visual content by iterative denoising based on the new prompt text using the pre-trained diffusion model.

[0211] Optionally, the generation module 804 is further configured to: determine a new prompt text using a pre-trained large language model, based at least on the evaluation information, the reconstructed prompt text, and the original prompt text.

[0212] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0213] Figure 9 This is a block diagram illustrating an electronic device 900 according to an exemplary embodiment. For example... Figure 9 As shown, the electronic device 900 may include a processor 901 and a memory 902. The electronic device 900 may also include one or more of a multimedia component 903, an input / output (I / O) interface 904, and a communication component 905.

[0214] The processor 901 controls the overall operation of the electronic device 900 to complete all or part of the steps in the aforementioned visual content generation method. The memory 902 stores various types of data to support the operation of the electronic device 900. This data may include, for example, instructions for any application or method operating on the electronic device 900, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 903 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 902 or transmitted via communication component 905. The audio component also includes at least one speaker for outputting audio signals. I / O interface 904 provides an interface between processor 901 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 905 is used for wired or wireless communication between the electronic device 900 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of these. Therefore, the corresponding communication component 905 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0215] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the visual content generation method described above.

[0216] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the visual content generation method described above. For example, the computer-readable storage medium may be the memory 902 including the program instructions described above, which may be executed by the processor 901 of the electronic device 900 to complete the visual content generation method described above.

[0217] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the visual content generation method described above.

[0218] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the visual content generation method described above.

[0219] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0220] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0221] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for generating visual content, characterized in that, include: Obtain the original visual content and the original prompt text used to generate the original visual content; A first reverse text is determined based on the original visual content using a pre-trained multimodal model. The first reverse text is used to describe the original visual content. Based on the first reversed text, the original prompt text is reconstructed to obtain the reconstructed prompt text; The target visual content is generated based on the reconstructed prompt text using a pre-trained diffusion model. The step of generating target visual content based on the reconstructed prompt text using a pre-trained diffusion model includes: performing iterative noise reduction on the reconstructed prompt text using the pre-trained diffusion model to generate visual content; controlling the iterative noise reduction of the pre-trained diffusion model during the process of generating visual content by iterative noise reduction on the reconstructed prompt text using the pre-trained diffusion model; and determining the visual content generated by the pre-trained diffusion model under the condition of stopping iterative noise reduction as the target visual content when the pre-trained diffusion model meets the stopping iterative noise reduction condition. The iterative denoising control of the pre-trained diffusion model includes: responding to the current iteration number of the pre-trained diffusion model reaching a preset iteration number, evaluating the iterative denoising of the pre-trained diffusion model to obtain evaluation information, wherein the evaluation information is used to characterize whether the iterative denoising of the pre-trained diffusion model is normal, abnormal, or excellent; and performing iterative denoising control on the pre-trained diffusion model based on the evaluation information. The evaluation of iterative denoising of the pre-trained diffusion model to obtain evaluation information includes: acquiring intermediate visual content generated by the pre-trained diffusion model at the current iteration number and historical visual content generated at iteration numbers prior to the current iteration number; determining the quality of the intermediate visual content based on the intermediate visual content, the reconstruction prompt text, and the historical visual content; determining the semantic matching degree between the reconstruction prompt text and the intermediate visual content; determining the potential deviation risk of the intermediate visual content based on the quality of the intermediate visual content and the semantic matching degree; and determining evaluation information based on the quality of the intermediate visual content and the potential deviation risk.

2. The visual content generation method according to claim 1, characterized in that, The step of reconstructing the original prompt text based on the first reverse text to obtain the reconstructed prompt text includes: The quality of the original visual content is determined by using a pre-trained large language model based on the original prompt text and the first reverse text. Based on the quality of the original visual content, the original prompt text is reconstructed to obtain the reconstructed prompt text.

3. The visual content generation method according to claim 2, characterized in that, The pre-trained large language model determines the quality of the original visual content based on the original prompt text and the first reverse text, including: The semantic similarity between the original prompt text and the first reverse text is determined by a pre-trained large language model. Using a pre-trained large language model, the matching degree is determined based on the original prompt text and the first reverse text. The matching degree is used to characterize the matching degree between the original prompt text and the original visual content. The quality of the original visual content is determined by using a pre-trained large language model based on the semantic similarity and the matching degree.

4. The visual content generation method according to claim 2, characterized in that, The step of reconstructing the original prompt text based on the quality of the original visual content to obtain the reconstructed prompt text includes: If the quality of the original visual content does not meet the preset quality requirements, a pre-trained large language model is used to determine the text to be added corresponding to the original prompt text, and to determine the text to be modified included in the original prompt text. The text to be modified includes redundant text and text unrelated to the original visual content. Based on the text to be added and the text to be modified, the original prompt text is reconstructed to obtain the reconstructed prompt text.

5. The visual content generation method according to claim 1, characterized in that, Determining the quality of the intermediate visual content based on the intermediate visual content, the reconstructed prompt text, and the historical visual content includes: Determine the first semantic similarity between the intermediate visual content and the reconstructed prompt text; Determine the second semantic similarity between the historical visual content and the reconstructed prompt text; Based on the first semantic similarity and the second semantic similarity, determine the semantic similarity change information; Determine the visual similarity between the intermediate visual content and the historical visual content; The quality of the intermediate visual content is determined based on the first semantic similarity, the semantic similarity change information, and the visual similarity.

6. The visual content generation method according to claim 1, characterized in that, Determining the semantic matching degree between the reconstructed prompt text and the intermediate visual content includes: A second reverse text is determined based on the intermediate visual content using a pre-trained multimodal model. The second reverse text is used to describe the intermediate visual content. Determine the semantic matching degree between the reconstructed prompt text and the second reverse text; The semantic matching degree between the reconstructed prompt text and the intermediate visual content is determined based on the semantic matching degree between the reconstructed prompt text and the second reverse text.

7. The visual content generation method according to claim 6, characterized in that, Determining the semantic matching degree between the reconstructed prompt text and the second reverse text includes: From the reconstruction prompt text, determine the first text used to describe the entities included in the visual content to be generated, the second text used to describe the attributes of the visual content to be generated, and the third text used to describe the spatial relationships included in the visual content to be generated. From the second reverse text, a fourth text for describing the entities included in the intermediate visual content, a fifth text for describing the attributes of the intermediate visual content, and a sixth text for describing the spatial relationships included in the intermediate visual content are determined. Determine the entity semantic matching degree based on the first text and the fourth text; Determine the semantic matching degree of the attributes based on the second text and the fifth text; Determine the semantic matching degree of spatial relationships based on the third text and the sixth text; The semantic matching degree between the reconstructed prompt text and the second reverse text is determined based on the entity semantic matching degree, the attribute semantic matching degree, and the spatial relationship semantic matching degree.

8. The visual content generation method according to claim 1, characterized in that, The step of iteratively denoising the pre-trained diffusion model based on the evaluation information includes: If the evaluation information indicates that the pre-trained diffusion model is performing normally in the iterative denoising, the pre-trained diffusion model is controlled to continue the next iteration of denoising. If the evaluation information indicates that the pre-trained diffusion model has excellent iterative denoising performance, the pre-trained diffusion model is controlled to continue the next iteration of denoising, and the frequency of iterative denoising control of the pre-trained diffusion model is reduced.

9. The visual content generation method according to claim 1, characterized in that, The step of iteratively denoising the pre-trained diffusion model based on the evaluation information includes: If the evaluation information indicates that the pre-trained diffusion model is abnormal in iterative denoising, the pre-trained diffusion model is controlled to stop iterative denoising based on the reconstructed prompt text; Determine the new prompt text; The pre-trained diffusion model is used to iteratively reduce noise based on the new prompt text to generate visual content. During the process of generating visual content by iteratively denoising the pre-trained diffusion model based on the new prompt text, the pre-trained diffusion model is subjected to iterative denoising control again.

10. The visual content generation method according to claim 9, characterized in that, The process of determining the new prompt text includes: Using a pre-trained large language model, a new prompt text is determined based at least on the evaluation information, the reconstructed prompt text, and the original prompt text.

11. A visual content generation device, characterized in that, include: The acquisition module is used to acquire the original visual content and the original prompt text used to generate the original visual content; The determination module is used to determine a first reverse text based on the original visual content using a pre-trained multimodal model. The first reverse text is used to describe the original visual content. The reconstruction module is used to reconstruct the original prompt text based on the original prompt text and the first reverse text to obtain the reconstructed prompt text; The generation module is used to generate target visual content based on the reconstructed prompt text using a pre-trained diffusion model; The generation module is further configured to: generate visual content by iteratively denoising the reconstructed prompt text using the pre-trained diffusion model; control the iterative denoising of the pre-trained diffusion model during the process of generating visual content by iteratively denoising the reconstructed prompt text using the pre-trained diffusion model; and determine the visual content generated by the pre-trained diffusion model under the condition of stopping iterative denoising as the target visual content when the pre-trained diffusion model meets the stopping iterative denoising condition. The generation module is further configured to: in response to the current iteration number of the pre-trained diffusion model reaching a preset iteration number, evaluate the iterative denoising of the pre-trained diffusion model to obtain evaluation information, wherein the evaluation information is used to characterize whether the iterative denoising of the pre-trained diffusion model is normal, abnormal, or excellent; and perform iterative denoising control on the pre-trained diffusion model based on the evaluation information. The generation module is further configured to: obtain the intermediate visual content generated by the pre-trained diffusion model at the current iteration number and the historical visual content generated at iteration numbers prior to the current iteration number; and determine the quality of the intermediate visual content based on the intermediate visual content, the reconstructed prompt text, and the historical visual content. Determine the semantic matching degree between the reconstructed prompt text and the intermediate visual content; determine the potential deviation risk of the intermediate visual content based on the quality of the intermediate visual content and the semantic matching degree; determine evaluation information based on the quality of the intermediate visual content and the potential deviation risk.

12. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the visual content generation method according to any one of claims 1-10.

Images