Content generation method and apparatus

Abstract

Embodiments of the present application disclose a content generation method and device. The main technical scheme comprises: obtaining input data, the input data at least comprising a first control condition; using a first content generation model, based on a guide weight corresponding to each time step in a pre-stored guide weight curve, performing denoising processing on a noisy media content to generate a first target media content corresponding to the first control condition; wherein the first content generation model adopts a classifier-free guided diffusion model, the guide weight is a weight corresponding to a guide vector in the process of predicting noise, and the guide vector is a difference between a first noise vector predicted based on the first control condition and a second noise vector predicted under a non-prompt condition. The present application can improve the quality of media content generation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a content generation method and device. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI-generated content (AIGC) has achieved significant breakthroughs in recent years. In particular, diffusion-based text-to-image and text-to-video technologies have become a hot topic in research and application due to their superior quality and rich diversity. Diffusion models, through a gradual denoising process, can transform an initial pure noise tensor into an image or video content that matches the user's text description.

[0003] In existing diffusion models, classifier-free guidance (CFG) techniques are widely used to enhance the ability of generated content to adhere to textual instructions. This technique guides the generation process by interpolating between unconditional and conditional predictions, allowing the generated image or video to more closely align with the user's textual prompt. However, existing CFG schemes generally rely on fixed guidance weights, which are typically set to fixed values ​​or pre-set through manual experience. This fixed weight approach can significantly impact the quality of generated content. Summary of the Invention

[0004] In view of this, the present application provides a content generation method and apparatus to improve the quality of media content generation.

[0005] This application provides the following solutions:

[0006] In a first aspect, a content generation method is provided, the method comprising:

[0007] Acquiring input data, wherein the input data includes at least a first control condition;

[0008] Using the first content generation model, the noisy media content is denoised based on the guidance weight corresponding to each time step in the pre-stored guidance weight curve to generate the first target media content corresponding to the first control condition; wherein, the first content generation model adopts a diffusion model without classifier guidance, the guidance weight is the weight corresponding to the guidance vector when predicting noise in the denoising process, and the guidance vector is the difference between the first noise vector predicted based on the first control condition and the second noise vector predicted under the no-prompt condition.

[0009] Optionally, the first content generation model comprises a conditional model and an unconditional model.

[0010] The denoising of the noisy media content is performed based on a pre-stored guide weight curve corresponding to each time step.

[0011] At each time step in the denoising process, the first noise vector is predicted based on the first control condition by using the conditional model, the second noise vector is predicted by using the unconditional model, a guide weight corresponding to the current time step is determined based on the guide weight curve, the first noise vector and the guide vector are weighted by using the guide weight to obtain a predicted noise corresponding to the current time step, and the guide vector is a difference between the first noise vector and the second noise vector.

[0012] The denoising of the noisy media content is performed based on the predicted noise corresponding to each time step.

[0013] Optionally, the conditional model and the unconditional model satisfy at least one of the following asymmetric conditions:

[0014] The conditional model is an enhanced model fine-tuned for a specific field, and the unconditional model is a base model not fine-tuned for the specific field.

[0015] The parameter amount of the conditional model is greater than that of the unconditional model.

[0016] The structural complexity of the conditional model is higher than that of the unconditional model.

[0017] Optionally, the pre-stored guide weight curve is constructed in the following manner:

[0018] A curve function controlled by an adjustable parameter set is determined, and the curve function is used to define the mapping relationship between time steps and guide weights.

[0019] A test sample set is obtained.

[0020] Based on the test sample set, media generation effect evaluations are performed on candidate curve functions obtained by using different value combinations of the parameter set, and media generation effect evaluation results corresponding to each value combination are obtained.

[0021] Based on the media generation effect evaluation results, a curve function with the optimal media evaluation effect is selected from the candidate curve functions.

[0022] The curve function with the optimal media evaluation effect selected is stored as the guide weight curve.

[0023] Optionally, the determining the curve function controlled by the adjustable parameter set comprises:

[0024] determining a curve function controlled by an adjustable first guide weight parameter, a second guide weight parameter, a third guide weight parameter and a target point time step parameter; wherein the first guide weight parameter corresponds to a starting point time step in a denoising process, the second guide weight parameter corresponds to the target point time step parameter in the denoising process, and the third guide weight parameter corresponds to an ending point time step in the denoising process; the second guide weight parameter is a guide weight peak value in the curve function, the first guide weight parameter and the third guide weight parameter are both less than the second guide weight parameter, and the target point time step is an intermediate point time step between the starting point time step and the ending point time step.

[0025] Optionally, the test sample set comprises at least one second control condition;

[0026] The media generation effect evaluation result corresponding to each value combination is obtained by performing media generation effect evaluation on the candidate curve functions obtained by using different value combinations of the parameter set based on the test sample set, comprising:

[0027] For each value combination of the parameter set, the following is performed:

[0028] each second target media content corresponding to the second control condition is generated by using the candidate curve function obtained by using the current value combination and the second content generation model;

[0029] objective index values and / or subjective scores of each second target media content are determined;

[0030] Based on the objective index values and / or subjective scores of each second target media content, a comprehensive evaluation score of the current value combination is determined;

[0031] The curve function with the optimal media evaluation effect is selected from the candidate curve functions based on the media generation effect evaluation result, comprising:

[0032] The curve function corresponding to the value combination with the highest comprehensive evaluation score is selected from the candidate curve functions as the curve function with the optimal media evaluation effect.

[0033] Optionally, the objective index values comprise:

[0034] at least one of saturation, contrast and color deviation of the second target media content;

[0035] matching degree between the second target media content and the corresponding second control condition.

[0036] In a second aspect, a content generation device is provided, the device comprising:

[0037] a data acquisition unit configured to acquire input data, wherein the input data includes a first control condition;

[0038] The content generation unit is configured to use a first content generation model to denoise the noisy media content based on the guidance weight corresponding to each time step in a pre-stored guidance weight curve, so as to generate first target media content corresponding to the first control condition; wherein, the first content generation model adopts a diffusion model without classifier guidance, the guidance weight is the weight corresponding to the guidance vector when predicting noise during the denoising process, and the guidance vector is the difference between the first noise vector predicted based on the first control condition and the second noise vector predicted under the no-prompt condition.

[0039] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed, the steps of the method of the first aspect are implemented.

[0040] In a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method of the first aspect above.

[0041] In a fifth aspect, a computer program product is provided, comprising a computer program, which implements the steps of the method of the first aspect when executed.

[0042] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0043] 1) After obtaining the first control condition, this application utilizes the first content generation model and, based on a pre-stored guidance weight curve, accurately determines the guidance weight corresponding to each time step, thereby generating the first target media content corresponding to the first control condition. This application introduces a dynamic guidance mechanism to dynamically adjust the guidance weights. This process not only ensures that the generated target media content conforms to the text prompt but also effectively enhances the authenticity and naturalness of the content, thereby significantly improving the quality of content generation.

[0044] 2) The first content generation model in the embodiment of the present application adopts an asymmetric model architecture of conditional models and unconditional models. Through the design of this asymmetric model architecture, the guidance vector becomes more significant and stable, so that a significant guidance effect can be obtained even at a lower guidance weight value, effectively improving the guidance efficiency while ensuring the authenticity of the generated content and a high degree of fit with the control conditions.

[0045] 3) The embodiment of the present application constructs a curve function controlled by an adjustable parameter set, determines the optimal curve function by adjusting the parameter set in the curve function, and uses the optimal curve function as a guiding weight curve, so that the guiding weight can dynamically change with time steps, thereby providing more accurate guiding weights for denoising processing, improving the overall media generation quality, and ensuring that the generated content achieves the best balance in terms of structural stability, color authenticity, and detail expression.

[0046] 4) This embodiment of the present application controls the curve function through adjustable first, second, and third guiding weight parameters, as well as the target point time step parameter. This parameterized approach makes the curve function construction more flexible and controllable, allowing parameters to be adjusted according to different denoising process requirements to achieve fine-grained control of the guiding weight curve, better adapting to various media generation tasks.

[0047] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of a system architecture applicable to an embodiment of the present application.

[0050] Figure 2 A flowchart of a content generation method provided in an embodiment of the present application.

[0051] Figure 3 A schematic block diagram of a content generation device provided in an embodiment of the present application.

[0052] Figure 4 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0054] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0055] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0056] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0057] In existing CFG technology, the core idea is to guide the generation process in a direction that better meets the specified conditions by combining conditional prediction (based on text prompts) and unconditional prediction (without any prompts) at each time step of the denoising process. Specifically, at each time step t, the calculation formula for the predicted noise vpred is as follows:

[0058] vpred = vuncond-wcfg×(vcond-vuncond)

[0059] Where vpred is the predicted noise at each time step t;

[0060] vuncond is the model's prediction result for the current noisy image without text prompts, which tends to generate an "average" and high-probability image.

[0061] vcond is the model's prediction result for the current noisy image under the condition of text prompts, which contains the semantic information of the text prompts.

[0062] (vcond - vuncond) is the guidance vector, pointing in the direction of the image that "fits the text cue".

[0063] wcfg is the guidance scale, a fixed global hyperparameter set by the user or developer to control the guidance strength.

[0064] While larger wcfg values ​​can enhance the effect of guidance vectors, making the generated content more closely align with text prompts, excessively increasing wcfg values ​​can compromise image realism, resulting in oversaturated colors, unnatural textures, unusual details, and texture breakdown—known as color anomalies or artifacts. Conversely, lowering the wcfg value to avoid these issues improves naturalness but weakens responsiveness to text prompts. Therefore, a fixed wcfg value makes it difficult to simultaneously achieve both generated image realism and high-fidelity responsiveness to text prompts.

[0065] In view of this, the present application provides a new approach. To facilitate understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 This is a schematic diagram of the system architecture applicable to the embodiments of the present application, such as Figure 1 As shown in , the system architecture may include: a user device, a content generation device located on the server side, and a device for training a content generation model.

[0066] The user equipment and the server can communicate with each other. The user equipment and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0067] User devices include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices include smart TVs and smart refrigerators. Wearable devices include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0068] A server can be a standalone server, a server cluster, or even a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0069] Before performing a content generation task, the device for training a content generation model can train the content generation model.

[0070] Users can input data through their user devices, which then transmit the input data over the network to a content generation device on the server side. The content generation device uses a trained content generation model to generate target media content and returns this target media content to the user device over the network. The user device then displays the received target media content to the user.

[0071] Apart from Figure 1 In addition to the illustrated architecture, a computer terminal device with strong computing capabilities may also generate target media content using the method provided in the embodiments of the present application.

[0072] It should be understood that Figure 1 The number of user devices, apparatuses for training content generation models, content generation apparatuses, and content generation models in the embodiment is merely illustrative. Any number of user devices, apparatuses for training content generation models, content generation apparatuses, and content generation models may be provided as needed.

[0073] It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have restrictions on size, order and quantity, but are only used to distinguish in name. For example, "first control condition" and "second control condition" are used to distinguish two types of text prompt data, and "first target media content" and "second target media content" are used to distinguish two types of media content. "First content generation model" and "second content generation model" are used to distinguish two content generation models, and so on.

[0074] Figure 2 This is a flowchart of a content generation method provided in an embodiment of the present application. The method can be Figure 1 The content generation device in the system shown is executed. Figure 2 As shown in , the method may include the following steps:

[0075] Step 201: Acquire input data, where the input data at least includes a first control condition.

[0076] Step 202: Using the first content generation model, based on the guidance weight corresponding to each time step in the pre-stored guidance weight curve, the noisy media content is denoised to generate the first target media content corresponding to the first control condition; wherein the first content generation model adopts a diffusion model without classifier guidance, the guidance weight is the weight corresponding to the guidance vector when predicting the noise during the denoising process, and the guidance vector is the difference between the first noise vector predicted based on the first control condition and the second noise vector predicted under the no-prompt condition.

[0077] As can be seen from the above process, after obtaining the first control condition, the first content generation model is used, and based on the pre-stored guide weight curve, the guide weight corresponding to each time step is accurately determined, and then the first target media content corresponding to the first control condition is generated. By introducing the dynamic guide mechanism, the dynamic adjustment of the guide weight is realized, which not only ensures that the generated target media content meets the text prompt, but also effectively enhances the authenticity and naturalness of the content, thereby significantly improving the content generation quality.

[0078] The steps in the above process and the effects that can be further produced will be described in detail below in conjunction with embodiments.

[0079] First, the step 201, i.e., "obtaining input data, the input data including a first control condition", will be described in detail in conjunction with an embodiment.

[0080] The content generation method provided in the embodiments of the present application can be applied to various content generation tasks, such as image generation tasks, video generation tasks, etc.

[0081] In the above content generation task, the input data can include a first control condition, where the first control condition is usually content input or selected by a user, such as text prompt data or image prompt data, etc. Taking the text prompt data as an example, the text prompt data mainly involves semantic understanding and description of the generated content, such as describing the theme, plot, action, scene, etc. of the generated video, so as to provide clear guidance for video generation.

[0082] In addition to the first control condition, the input data can also include noise media content. These noise data can be pre-generated random noise data, and the first content generation model subsequently performs denoising processing on the noise media content while combining the first control condition, to finally obtain the first target media content corresponding to the first control condition.

[0083] Next, the step 202, i.e., "using a first content generation model to perform denoising processing on the noise media content based on the guide weight corresponding to each time step in the pre-stored guide weight curve, to generate the first target media content corresponding to the first control condition", will be described in detail in conjunction with an embodiment.

[0084] The first content generation model adopts a classifier-free guided diffusion model, the guide weight is the weight of the guide vector corresponding to the prediction of noise in the denoising process, and the guide vector is the difference between the first noise vector predicted based on the first control condition and the second noise vector predicted under the condition of no prompt.

[0085] In the embodiments of the present application, the guide weight curve can be a dynamic function curve wcfg(t) about the time step t, which can be pre-stored in the first content generation model.

[0086] Taking an image generation task as an example, assume that the first control condition is “a golden retriever running on the grass, sunny, and colorful picture”. When generating the image, first, the noise media content is generated. Then, in the denoising process, for each time step t, the corresponding guide weight w cfg (t) is obtained according to the pre-stored guide weight curve. Based on this guide weight, the guide vector is calculated, which is obtained by subtracting the second noise vector predicted under the condition of no prompt from the first noise vector predicted based on the text prompt data “a golden retriever running on the grass, sunny, and colorful picture”. In the denoising process, the guide vector is weighted by the guide weight w cfg (t), so that the noise media content is directionally denoised, gradually approaching the image that meets the description of the first control condition, and finally generating the image corresponding to the first control condition, that is, the first target media content.

[0087] As an implementable manner, the guide weight curve in the embodiment of the present application can be constructed in the following manner:

[0088] A curve function controlled by an adjustable parameter set is determined, which is used to define the mapping relationship between the time step and the guide weight;

[0089] A test sample set is obtained;

[0090] Based on the test sample set, the media generation effect of the candidate curve function obtained by using different value combinations of the parameter set is evaluated, and the media generation effect evaluation result corresponding to each value combination is obtained;

[0091] Based on the media generation effect evaluation result corresponding to each value combination, the curve function with the optimal media evaluation effect is selected from the candidate curve functions;

[0092] The selected curve function is used as the guide weight curve and is stored.

[0093] This manner determines the optimal curve function by constructing a curve function controlled by an adjustable parameter set and adjusting the parameter set in the curve function, and uses the optimal curve function as the guide weight curve, so that the guide weight can dynamically change with the time step, thereby providing more accurate guide weight for the denoising process, improving the overall media generation quality, and ensuring that the generated content achieves the best balance in terms of structural stability, color authenticity, and detail performance.

[0094] Specifically, we first need to define a curve function controlled by a set of adjustable parameters. To this end, we conducted an in-depth analysis of the denoising process and found that it can be divided into three key stages: the initial denoising phase, the mid-stage denoising phase, and the final denoising phase. Each stage exhibits significant differences in its response to adjustments to the guidance weight (wcfg value):

[0095] In the early stages of denoising, this phase focuses on building the basic structure and outline of the generated content. Appropriately increasing the wcfg value at this stage can significantly improve the structural stability of the generated content, but it can also lead to color anomalies and a sharp increase in saturation. In the mid-stage of denoising, this phase focuses on refining the structure of the generated content. Appropriately increasing the wcfg value at this stage can stabilize the structure and dynamics of the generated content with minimal negative impact on color. In the late stages of denoising, this phase focuses on building the details and texture of the generated content. Increasing the wcfg value has little effect on the structure of the generated content and can even introduce unnatural local highlights or bright spots, disrupting the image's texture.

[0096] Based on the above analysis, it is found that an ideal guidance weight curve should have an asymmetric distribution characteristic of "high in the middle stage, low in the early and late stages", that is, strong guidance is applied in the middle stage of denoising when the core structure and content are constructed, and weak guidance is applied in the early and late stages of denoising when color distortion and bright spots are prone to occur.

[0097] Therefore, in the embodiment of the present application, the undetermined coefficient method can be used to define a curve function through four adjustable parameters, that is, a curve function controlled by an adjustable first guide weight parameter, a second guide weight parameter, a third guide weight parameter and a target point time step parameter; wherein, the first guide weight parameter corresponds to the starting point time step in the denoising process, the second guide weight parameter corresponds to the target point time step parameter in the denoising process, and the third guide weight parameter corresponds to the ending point time step in the denoising process; the second guide weight parameter is the guide weight peak in the curve function, the first guide weight parameter and the third guide weight parameter are both smaller than the second guide weight parameter, and the target point time step is the midpoint time step between the starting point time step and the ending point time step.

[0098] The starting time step refers to the initial state of the denoising process, in which case we define time step t=1. The ending time step refers to the final state of the denoising process, in which case we define time step t=0. During the denoising process, time step t decreases gradually from t=1 to t=0. In other words, the smaller the t value, the closer the denoising process is to completion. The target time step refers to the intermediate state between the start and end of the denoising process, in which case t=k, where k is between 0 and 1.

[0099] Specifically, at t = 1, set wcfg(1) = w1;

[0100] At t = k, set wcfg(k) = w2, where w2 is the peak guide weight in the curve function;

[0101] At t=0, set wcfg(0)=w3.

[0102] Among them, the above w1 is the first guided weight parameter, w2 is the second guided weight parameter, w3 is the third guided weight parameter, k is the target point time step parameter, and w1 and w3 are both smaller than w2.

[0103] Using these four parameters, we can use two quadratic curves or higher-order spline curves to fit the function curve wcfg(t). For example, we can fit a quadratic function at t∈[k, 1] and t∈[0, k] to generate the curve function wcfg(t).

[0104] Based on different value combinations of the above four parameters, multiple candidate function curves wcfg(t) can be generated. The media generation effect of each candidate curve function is then evaluated through a test sample set. Finally, the curve function with the best media evaluation effect is selected as the guiding weight curve and stored in the first content generation model.

[0105] The present embodiment controls the curve function through adjustable first, second, and third guiding weight parameters, as well as the target point time step parameter. This parameterized approach makes the curve function construction more flexible and controllable, allowing parameters to be adjusted according to different denoising process requirements to achieve fine-grained control of the guiding weight curve, better adapting to various media generation tasks.

[0106] The test sample set may include at least one second control condition. This second control condition is typically user input or selection, such as text prompt data or image prompt data. This second control condition must cover the diverse generation requirements that users may encounter in real applications to fully verify the generalization capabilities of the candidate curve function.

[0107] Based on the test sample set, the process of evaluating the media generation effect of the candidate curve function may include but is not limited to:

[0108] For each value combination of the parameter set (w1, w2, w3, k), perform the following operations:

[0109] First, using the candidate curve function obtained from the current value combination and the second content generation model, second target media content corresponding to each second control condition is generated. The second content generation model can be generated based on a diffusion model. The candidate curve function obtained from the current value combination is applied to the second content generation model to generate the second target media content based on the second control condition.

[0110] Then, an objective indicator value and / or a subjective score of each second target media content is determined.

[0111] The objective indicator values ​​may include image quality indicators such as the saturation, contrast, and color cast of the second target media content, as well as the degree of match between the second target media content and the corresponding second control condition. For example, the saturation of the second target media content can be evaluated by calculating the variance of the pixel color channels to determine whether it approximates the natural distribution of a real image. The contrast of the second target media content can be determined based on the dynamic range of the image grayscale histogram. The color cast of the second target media content can be quantified by measuring the degree of color distortion using cosine similarity with a real-scene color library. The degree of match between the second target media content and the corresponding second control condition can be calculated using a pre-trained multimodal model to calculate the similarity between the embedding vectors of the generated content and the corresponding text prompt, thereby measuring the semantic responsiveness to the instruction. The subjective score can be obtained through manual evaluation based on the evaluator's subjective perception of the visual effects of the second target media content and its fit with the second control condition.

[0112] Then, based on the objective indicator values ​​and / or subjective scores of each second target media content, the comprehensive evaluation score of the current value combination is determined. The comprehensive evaluation score can be obtained by weighted calculation of the objective indicator values ​​and subjective scores, etc., to fully reflect the overall effect of the candidate curve function corresponding to the current value combination in media generation. For example, each value combination can obtain corresponding multiple second target media contents. First, based on the objective indicator values ​​and / or subjective scores of each second target media content, the comprehensive evaluation score of each second target media content is determined, and then the average value is taken to obtain the comprehensive evaluation score corresponding to the current value combination.

[0113] Finally, from the candidate curve functions, the curve function corresponding to the value combination with the highest comprehensive evaluation score is selected as the curve function with the best media evaluation effect, and is stored as the guide weight curve.

[0114] The curve function selected in this way can effectively avoid problems such as color distortion and bright spots while ensuring the stability of the generated content structure, improve the overall media generation quality, and better meet the diverse media generation needs in practical applications.

[0115] It should also be noted that the embodiments of the present application have flexible and diverse ways of constructing the guidance weight curve. In addition to constructing the guidance weight curve through piecewise quadratic functions, other functions can also be used to define the guidance weight curve, such as Gaussian functions, bell curves, etc. In addition, a neural network, such as a multi-layer perceptron, can be used, with its input being the time step t and its output being the corresponding wcfg value, and the neural network can be trained to learn the appropriate guidance weight curve. In short, as long as a dynamic guidance weight curve that meets the needs can be constructed, the embodiments of the present application do not impose specific restrictions on this.

[0116] Furthermore, to further enhance the effectiveness of the guidance vector (vcond - vuncond) and improve the guidance effect of the guidance weight wcfg, the first content generation model in the embodiment of the present application can adopt an asymmetric model architecture, specifically including a conditional model and an unconditional model. On this basis, based on the guidance weight corresponding to each time step in the pre-stored guidance weight curve, denoising of noisy media content can be achieved as follows:

[0117] Executed at each time step of the denoising process:

[0118] First, the conditional model is used to predict the first noise vector (vcond in the above formula) based on the first control condition (such as text prompt data). The unconditional model is then used to predict the second noise vector (vuncond in the above formula). Next, the guidance weight corresponding to the current time step is determined based on the guidance weight curve. The first and second noise vectors are weighted using the determined guidance weight to obtain the predicted noise corresponding to the current time step, where the guidance vector is the difference between the first and second noise vectors (vcond - vuncond).

[0119] Finally, based on the predicted noise corresponding to each time step, the noisy media content is denoised.

[0120] In order to implement the asymmetric model architecture of the first content generation model, the conditional model and the unconditional model in the embodiment of the present application need to meet at least one of the following asymmetric conditions:

[0121] The conditional model is an enhanced model that is fine-tuned for a specific domain, while the unconditional model is a base model that is not fine-tuned for a specific domain. This difference makes the conditional model more advantageous when dealing with certain types of control conditions, and it can generate conditional predictions that are more consistent with domain characteristics.

[0122] Conditional models have more parameters than unconditional models. More parameters enable conditional models to capture and express more complex features and patterns, thus providing richer details and more accurate semantic information in conditional predictions.

[0123] Conditional models have higher structural complexity than unconditional models. For example, conditional models can add more parameters or computational modules to certain network layers (such as the cross-attention layer) to make them more capable of processing conditional information, while unconditional models maintain a relatively simple structure and focus on generating generalized noisy predictions.

[0124] Through the design of this asymmetric model architecture, the guidance vector becomes more significant and stable, so that the embodiment of the present application can achieve significant guidance effects even at lower guidance weight values, effectively improving the guidance efficiency while ensuring the authenticity of the generated content and a high degree of fit with the control conditions.

[0125] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0126] According to an embodiment of another aspect, a content generating apparatus is provided. Figure 3 This is a schematic block diagram of the device provided in the embodiment of the present application. Figure 3 As shown, the device 300 mainly includes: a data acquisition unit 301 and a content generation unit 302. The main functions of each component unit are as follows:

[0127] The data acquisition unit 301 is configured to acquire input data, where the input data includes a first control condition;

[0128] The content generation unit 302 is configured to use the first content generation model to denoise the noisy media content based on the guidance weight corresponding to each time step in the pre-stored guidance weight curve, so as to generate the first target media content corresponding to the first control condition; wherein, the guidance weight curve represents the guidance weight corresponding to each time step in the denoising process of the first content generation model, the first content generation model adopts a diffusion model without classifier guidance, the guidance weight is the weight corresponding to the guidance vector when predicting noise in the denoising process, and the guidance vector is the difference between the first noise vector predicted based on the first control condition and the second noise vector predicted under the no-prompt condition.

[0129] Optionally, the first content generation model includes a conditional model and an unconditional model;

[0130] The content generation unit 302 may be specifically configured to:

[0131] At each time step in the denoising process, the following steps are performed: using the conditional model and the first control condition to predict the first noise vector; using the unconditional model to predict the second noise vector; determining the guidance weight corresponding to the current time step based on the guidance weight curve; using the guidance weight, performing weighted processing on the first noise vector and the guidance vector to obtain the predicted noise corresponding to the current time step, where the guidance vector is the difference between the first noise vector and the second noise vector;

[0132] Based on the predicted noise corresponding to each of the time steps, denoising processing is performed on the noisy media content.

[0133] Optionally, the conditional model and the unconditional model satisfy at least one of the following asymmetric conditions:

[0134] The conditional model is an enhanced model that is fine-tuned for a specific domain, and the unconditional model is a basic model that is not fine-tuned for the specific domain;

[0135] The conditional model has a larger number of parameters than the unconditional model;

[0136] The structural complexity of the conditional model is higher than that of the unconditional model.

[0137] Optionally, the apparatus 300 may further include a curve construction unit 303 configured to:

[0138] Determining a curve function controlled by an adjustable parameter set, wherein the curve function is used to define a mapping relationship between time steps and guidance weights;

[0139] Get a test sample set;

[0140] Based on the test sample set, performing a media generation effect evaluation on candidate curve functions obtained by using different value combinations of the parameter set to obtain a media generation effect evaluation result corresponding to each value combination;

[0141] Based on the media generation effect evaluation result, screening out a curve function with the best media evaluation effect from the candidate curve functions;

[0142] The curve function with the best media evaluation effect is selected as the guiding weight curve and stored.

[0143] Optionally, the curve construction unit 303 determines a curve function controlled by an adjustable parameter set, which is specifically configured as follows:

[0144] Determine a curve function controlled by an adjustable first guide weight parameter, a second guide weight parameter, a third guide weight parameter and a target point time step parameter; wherein the first guide weight parameter corresponds to the starting point time step in the denoising process, the second guide weight parameter corresponds to the target point time step parameter in the denoising process, and the third guide weight parameter corresponds to the ending point time step in the denoising process; the second guide weight parameter is the guide weight peak in the curve function, the first guide weight parameter and the third guide weight parameter are both less than the second guide weight parameter, and the target point time step is the midpoint time step between the starting point time step and the ending point time step.

[0145] Optionally, the test sample set includes at least one second control condition;

[0146] The curve construction unit 303 performs a media generation effect evaluation on the candidate curve functions obtained by using different value combinations of the parameter set based on the test sample set, and obtains a media generation effect evaluation result corresponding to each value combination. The configuration may be specifically as follows:

[0147] For each value combination of the parameter set, execute:

[0148] Generate second target media content corresponding to each second control condition using the candidate curve function obtained by the current value combination and the second content generation model;

[0149] determining an objective indicator value and / or a subjective score for each of the second target media contents;

[0150] Determining a comprehensive evaluation score of the current value combination based on the objective indicator value and / or subjective score of each second target media content;

[0151] The step of selecting a curve function with the best media evaluation effect from the candidate curve functions based on the media generation effect evaluation result includes:

[0152] From the candidate curve functions, the curve function corresponding to the value combination with the highest comprehensive evaluation score is selected as the curve function with the best media evaluation effect.

[0153] Optionally, the objective indicator value includes:

[0154] at least one of saturation, contrast, and color cast of the second target media content;

[0155] The degree of matching between the second target media content and the corresponding second control condition.

[0156] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0158] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0159] And an electronic device comprising:

[0160] one or more processors; and

[0161] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0162] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed.

[0163] in, Figure 4The schematic block diagram of an electronic device provided in an embodiment of the present application may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, and memory 420 may be communicatively connected via a communication bus 430. The input / output interface 413 may also be referred to as an I / O interface 413.

[0164] The processor 410 may be implemented as a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and may be used to execute relevant programs to implement the technical solutions provided in this application.

[0165] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operations of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and a content generation device 300, etc. can also be stored. The above-mentioned content generation device 300 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0166] The input / output interface 413 is used to connect to input / output modules to enable information input and output. The input / output modules can be configured as components within the device (not shown) or externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, and various sensors, while output devices may include a display, speaker, vibrator, indicator light, and the like.

[0167] The network interface 414 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.).

[0168] The bus 430 comprises a pathway for transmitting information between the various components of the device, such as the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 .

[0169] It should be noted that although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0170] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product. The computer program product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0171] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A content generation method, characterized in that: The method comprises: Acquiring input data, wherein the input data includes at least a first control condition; Using a first content generation model, based on the guidance weight corresponding to each time step in a pre-stored guidance weight curve, denoising is performed on the noisy media content to generate first target media content corresponding to the first control condition; wherein the first content generation model adopts a diffusion model without classifier guidance, the guidance weight is the weight corresponding to the guidance vector when predicting noise during the denoising process, the guidance vector is the difference between a first noise vector predicted based on the first control condition and a second noise vector predicted under the no-prompt condition, and the guidance weight curve is a dynamic function curve with respect to time steps; The first content generation model includes a conditional model and an unconditional model; The denoising process is performed on the noisy media content based on the guidance weight corresponding to each time step in the pre-stored guidance weight curve, including: At each time step in the denoising process, the following steps are performed: using the conditional model and the first control condition to predict the first noise vector; using the unconditional model to predict the second noise vector; determining the guidance weight corresponding to the current time step based on the guidance weight curve; and using the guidance weight to perform weighted processing on the first noise vector and the guidance vector to obtain the predicted noise corresponding to the current time step; Based on the predicted noise corresponding to each of the time steps, denoising processing is performed on the noisy media content.

2. The method according to claim 1, characterized in that The conditional model and the unconditional model satisfy at least one of the following asymmetric conditions: The conditional model is an enhanced model that is fine-tuned for a specific domain, and the unconditional model is a basic model that is not fine-tuned for the specific domain; The conditional model has a larger number of parameters than the unconditional model; The structural complexity of the conditional model is higher than that of the unconditional model.

3. The method according to claim 1 or 2, characterized in that The pre-stored guidance weight curve is constructed as follows: Determining a curve function controlled by an adjustable parameter set, wherein the curve function is used to define a mapping relationship between time steps and guidance weights; Get a test sample set; Based on the test sample set, performing a media generation effect evaluation on candidate curve functions obtained by using different value combinations of the parameter set to obtain a media generation effect evaluation result corresponding to each value combination; Based on the media generation effect evaluation result, screening out a curve function with the best media evaluation effect from the candidate curve functions; The curve function with the best media evaluation effect is selected as the guiding weight curve and stored.

4. The method according to claim 3, characterized in that The determining of a curve function controlled by an adjustable parameter set includes: Determine a curve function controlled by an adjustable first guide weight parameter, a second guide weight parameter, a third guide weight parameter and a target point time step parameter; wherein the first guide weight parameter corresponds to the starting point time step in the denoising process, the second guide weight parameter corresponds to the target point time step parameter in the denoising process, and the third guide weight parameter corresponds to the ending point time step in the denoising process; the second guide weight parameter is the guide weight peak in the curve function, the first guide weight parameter and the third guide weight parameter are both less than the second guide weight parameter, and the target point time step is the midpoint time step between the starting point time step and the ending point time step.

5. The method according to claim 3, characterized in that The test sample set includes at least one second control condition; The media generation effect evaluation is performed on the candidate curve functions obtained by using different value combinations of the parameter set based on the test sample set, and the media generation effect evaluation results corresponding to each value combination are obtained, including: For each value combination of the parameter set, execute: Generate second target media content corresponding to each second control condition using the candidate curve function obtained by the current value combination and the second content generation model; determining an objective indicator value and / or a subjective score for each of the second target media contents; Determining a comprehensive evaluation score of the current value combination based on the objective indicator value and / or subjective score of each second target media content; The step of selecting a curve function with the best media evaluation effect from the candidate curve functions based on the media generation effect evaluation result includes: From the candidate curve functions, the curve function corresponding to the value combination with the highest comprehensive evaluation score is selected as the curve function with the best media evaluation effect.

6. The method according to claim 5, characterized in that The objective indicator values ​​include: at least one of saturation, contrast, and color cast of the second target media content; The degree of matching between the second target media content and the corresponding second control condition.

7. A content generating device, characterized in that: The device comprises: a data acquisition unit configured to acquire input data, wherein the input data includes a first control condition; A content generation unit is configured to utilize a first content generation model to denoise noisy media content based on a guidance weight corresponding to each time step in a pre-stored guidance weight curve, thereby generating first target media content corresponding to the first control condition; wherein the first content generation model utilizes a diffusion model without classifier guidance, the guidance weight is a weight corresponding to a guidance vector when predicting noise during the denoising process, the guidance vector is a difference between a first noise vector predicted based on the first control condition and a second noise vector predicted under no prompt conditions, and the guidance weight curve is a dynamic function curve with respect to time steps; The first content generation model includes a conditional model and an unconditional model; The content generation unit is specifically configured to: At each time step in the denoising process, the following steps are performed: using the conditional model and the first control condition to predict the first noise vector; using the unconditional model to predict the second noise vector; determining the guidance weight corresponding to the current time step based on the guidance weight curve; and using the guidance weight to perform weighted processing on the first noise vector and the guidance vector to obtain the predicted noise corresponding to the current time step; Based on the predicted noise corresponding to each of the time steps, denoising processing is performed on the noisy media content.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images