Method, apparatus, device and storage medium for generating media content

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device and computer readable storage medium for generating media content. The method proposed herein comprises: receiving a media generation request; determining, by an encoding unit, a conditional feature corresponding to a first denoising time step based on the media generation request and a first noise representation of the first denoising time step; determining, by a decoding unit, a second noise representation of a second denoising time step based on the first noise representation and the conditional feature; and generating media content for the media generation request based on the second noise representation. Thus, embodiments of the present disclosure can improve the quality of the generated media content.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for generating media content. Background Technology

[0002] With the development of computer technology, generative models have been gradually applied to various types of generative tasks. For example, generative models, represented by diffusion models, can be applied to the process of generating media content. For instance, one can use text describing the media content to be generated to guide the diffusion model in generating the corresponding image or video content. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for generating media content is provided. The method includes: receiving a media generation request; using an encoding unit to determine a conditional feature corresponding to a first noise reduction time step based on the media generation request and a first noise representation of a first noise reduction time step; using a decoding unit to determine a second noise representation of a second noise reduction time step based on the first noise representation and the conditional feature; and generating media content for the media generation request based on the second noise representation.

[0004] In a second aspect of this disclosure, an apparatus for generating media content is provided. The apparatus includes: a receiving module configured to receive a media generation request; an encoding module configured to determine, using an encoding unit, a conditional feature corresponding to a first noise reduction time step based on the media generation request and a first noise representation of a first noise reduction time step; a decoding module configured to determine, using the decoding unit, a second noise representation of a second noise reduction time step based on the first noise representation and the conditional feature; and a generation module configured to generate media content for the media generation request based on the second noise representation.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 Examples of sample models for generating media content are shown, according to some embodiments of this disclosure;

[0011] Figure 3 A flowchart illustrating an example process for generating media content according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for generating media content according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] Traditional diffusion models require encoding low-frequency semantic information and decoding high-frequency details in each denoising step of media content generation. This necessitates reducing high-frequency components when encoding low-frequency semantics, while preserving them when decoding high-frequency details. This contradiction makes it difficult for the model to simultaneously optimize semantic extraction and detail recovery during training, resulting in slow training convergence and difficulty in further improving generation quality.

[0020] Embodiments of this disclosure propose a scheme for generating media content. The scheme includes: receiving a media generation request; using an encoding unit to determine a conditional feature corresponding to a first noise reduction time step based on the media generation request and a first noise representation of a first noise reduction time step; using a decoding unit to determine a second noise representation of a second noise reduction time step based on the first noise representation and the conditional feature; and generating media content for the media generation request based on the second noise representation.

[0021] In this way, by decoupling the noise reduction process into two stages, encoding and decoding, the embodiments of this disclosure can gradually recover the media content, thereby improving the quality of the generated media content.

[0022] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0023] Example Environment

[0024] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0025] like Figure 1 As shown, the electronic device 110 can receive a media generation request 120. As an example, the media generation request 120 may include a prompt message entered by the user. As an example, the prompt message may include text describing the media content to be generated, such as "a kitten".

[0026] In some embodiments, the media generation request 120 may also instruct any other appropriate generation parameters. For example, the media generation request 120 may instruct a reference image to generate media content based on any appropriate information such as the outline, style, skeleton, etc., of the reference image.

[0027] like Figure 1 As shown, electronic device 110 can utilize model 130 to process media generation request 120 to generate corresponding media content 140. Model 130 may include an appropriate generative model that supports the generation of media content. As an example, model 130 may include, for example, a diffusion model.

[0028] In some examples, the generated media content 140 may include, but is not limited to, image content, video content, audio content, or appropriate combinations thereof.

[0029] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example Model

[0033] Figure 2 An example model 200 for generating media content according to some embodiments of this disclosure is shown. Reference will be made below. Figure 2 This describes the specific process by which Model 200 generates the target video.

[0034] like Figure 2 As shown, model 200 includes an encoding unit 210 and a decoding unit 220. In some examples, the encoding unit 210 may include, for example, a conditional encoder. The decoding unit 220 may include, for example, a speed decoder for outputting a speed signal. Alternatively, the decoding unit 220 may support decoders for other noise reduction processes, such as an encoder that directly outputs the feature representation of the next noise reduction time step.

[0035] Compared to traditional diffusion models based on a single decoder, Model 200 can be divided into two parts: an encoding unit and a decoding unit. The encoding unit is specifically responsible for extracting low-frequency semantic information, while the decoding unit focuses on the recovery of high-frequency details. This decoupled design allows the model to extract semantic information and recover high-frequency details more efficiently during training.

[0036] The following will describe the specific process of generating media content using Model 200, taking into account the decoupled structure of Model 200.

[0037] Specifically, after model 200 receives the media generation request, for the first noise reduction time step t 204, encoding unit 210 can determine the first noise representation x corresponding to the first noise reduction time step t. t 202. Further, the coding unit 210 can be based on the first noise representation x. t The media generation request determines the conditional feature z corresponding to the first noise reduction time step. t 208.

[0038] Specifically, the media generation request can determine classification information associated with the media content to be generated. As an example, the classification information may include a classification label y 206 corresponding to the media content to be generated. For instance, if the media content to be generated is an image or video about a "kitten," the classification label y 206 could indicate the category "kitten."

[0039] Furthermore, a first noise representation x can be provided to the coding unit 210. t 202. First denoising time step t 204 and classification information (e.g., classification label y 206) to generate conditional feature z t 208. This process can be represented, for example, as:

[0040]

[0041] Furthermore, the generated conditional feature z t 208 can be injected into the decoding unit 220. Accordingly, the decoding unit 220 can base its decoding on the first noise representation x. t 202 and the conditional features of injection z t 208 generates a second noise representation corresponding to the next noise reduction time step (also known as the second noise reduction time step) to perform the noise reduction process.

[0042] In some examples, the decoding unit 220 may include, for example, a speed decoder. Figure 2 As shown, a first noise representation x can be provided to the decoding unit 220. t 202, First noise reduction time step t, 204, and conditional feature z t 208, to generate velocity features v corresponding to the first noise reduction time step t 204. t 212. This process can be represented, for example, as:

[0043] v t =Decoder(x) t ,t,z t (2)

[0044] Furthermore, x can be represented based on the first noise. t 202 and velocity characteristics vt 212, Determine the second noise representation for the second noise reduction time step. As an example, the process of determining the second noise representation can be expressed as:

[0045] x t-Δt =x t -Δt·v t (3)

[0046] Where Δt represents the difference between the second noise reduction time step and the first noise reduction time step.

[0047] In some embodiments, conditional feature z t 208 can be injected into the decoding unit 220 based on zero-initialization adaptive layer normalization (adaLN-Zero).

[0048] Specifically, the decoding unit 220 may include multiple blocks, each block may include interleaved attention networks and feedforward networks. Furthermore, each block contains normalization layers, the parameters of which can be obtained from the conditional features z. t 208 Dynamic Adjustment.

[0049] Specifically, conditional feature z t 208 can be mapped to a dimension that matches the decoder's feature space. This is typically achieved through a learnable projection function. Furthermore, the decoding unit 220 can be based on the mapped conditional features z. t 208 is used to dynamically determine the normalization parameters.

[0050] As an example, in the normalization layer of each decoder block, the mapped conditional feature z can be used. t 208. The normalization parameters are dynamically adjusted. The adjusted normalization parameters are used to normalize the features of the decoder, and the adjusted features are passed to the next layer of the decoder to continue participating in subsequent calculations, thereby realizing the noise reduction process of the noise features.

[0051] Therefore, conditional feature z t 208 can dynamically adjust the normalization parameters inside the decoder, thereby affecting the decoder's output. This is due to the conditional feature z. t 208 contains low-frequency semantic information. Through feature injection, embodiments of this disclosure can help the decoder better understand the semantic background of the current noise state, thereby generating high-frequency details more accurately.

[0052] Furthermore, embodiments of this disclosure allow for the use of conditional features z. t 208 dynamically adjusts the normalization parameters of the decoder. This dynamic adjustment can better adapt to feature changes at different time steps, improving the flexibility and adaptability of the model.

[0053] Furthermore, by injecting conditional features, the decoding unit 220 is able to better maintain the consistency of features across different time steps, thereby improving the quality of the generated media content.

[0054] Therefore, Model 200 can perform a denoising process corresponding to multiple denoising time steps based on the above process, thereby recovering the feature representation of the media content to be generated from the initial noise data, and thus generating the media content for the media generation request.

[0055] On the one hand, based on the decoupled design described above, the conditional encoder is specifically responsible for extracting low-frequency semantic information, while the speed decoder focuses on restoring high-frequency details. This decoupled design enables the model to extract semantic information and restore high-frequency details more efficiently during training.

[0056] On the other hand, by explicitly separating low-frequency and high-frequency processing, the embodiments of this disclosure are able to generate high-quality media content more accurately.

[0057] The training process of Model 200 will be further described below.

[0058] First, in order to ensure that the features generated by the coding unit 220 maintain local consistency across different time steps, a pre-trained visual model can be used to guide the training of the coding unit 220 during the training process.

[0059] Specifically, the training loss of coding unit 220 can be expressed as:

[0060]

[0061] According to formula (4), the first intermediate feature h determined by the encoding unit in processing the training generation request can be obtained. i Furthermore, the first intermediate feature h can be... i Mapped to reference feature r * The corresponding feature space is used to determine the second intermediate feature. (h i ). This represents a learnable mapping process, which can be implemented, for example, based on a multilayer perceptron (MLP).

[0062] Furthermore, reference features r can be generated based on the training media content corresponding to the training generation request. * For example, a pre-trained visual model can be used to encode training media content to generate reference features r. * .

[0063] Therefore, based on formula (4), we can base our work on the second intermediate feature. (h i ) and reference feature r* The difference was used to train encoding unit 220 to minimize the second intermediate feature. (h i ) and reference feature r * The differences.

[0064] For decoding unit 220, its training loss can be expressed as:

[0065]

[0066] Equation (5) represents the stream matching loss of the velocity decoder, the purpose of which is to train the decoder to accurately estimate the velocity signal, thereby gradually recovering the original data during the reverse process. Specifically, x data Represents the original data, ∈ represents the noise added during the diffusion process, v t This indicates the speed signal generated by the decoding unit.

[0067] In some embodiments, in order to improve the efficiency of model inference, the denoising process can also be sampled and optimized to reduce the computational cost of the encoding process.

[0068] Specifically, a preset set of time steps can be obtained, which may include multiple noise reduction time steps for the coding unit 210 to calculate conditional features.

[0069] Furthermore, in response to the noise reduction time step to be processed (e.g., the first noise reduction time step) matching the preset set of time steps, the encoding unit 210 can generate a conditional feature z corresponding to the noise reduction time step according to the process described above. t 208 (also known as the first condition feature).

[0070] Conversely, in response to a mismatch between the second noise reduction time step and a preset set of time steps, a second condition feature corresponding to the second noise reduction time step can be determined based on the first condition feature.

[0071] Specifically, the above sampling optimization process can be expressed as:

[0072]

[0073] Where Φ represents the preset set of time steps.

[0074] According to formula (6), if time step t matches Φ, the corresponding conditional feature z can be calculated using coding unit 210. t .

[0075] Conversely, if time step t does not match Φ, the time difference Δt between the first and second denoising time steps can be determined. Furthermore, this can be based on the already calculated conditional feature z. tThe time difference Δt is used to determine the second conditional feature corresponding to the second noise reduction time step.

[0076] Furthermore, in the case where time step t and Φ do not match, decoding unit 220 can further generate a third noise representation corresponding to the subsequent denoising time step (also known as the third denoising time step) based on the second conditional feature and the second noise representation determined by the time difference. Furthermore, encoding unit 210 and decoding unit 220 can further process the third noise representation to generate the final media content.

[0077] The following section will further explain the construction process of the preset time step set Φ.

[0078] After training model 200, the encoding unit 210 and decoding unit 220 in model 200 can be used to process a set of sample data to determine similarity information associated with multiple denoising time steps. The similarity information indicates the similarity of conditional features corresponding to different denoising time steps. As an example, the similarity information can be represented as the following similarity matrix S:

[0079] S[i,j]=cos(z) i , z j (7)

[0080] Furthermore, based on similarity information, a preset set of time steps can be determined from multiple denoising time steps using dynamic programming. Specifically, the goal of dynamic programming is to minimize the similarity loss:

[0081]

[0082] Specifically, in the dynamic programming process, cost information associated with candidate time steps can be determined based on similarity information. As an example, the cost information can be represented as the following cost matrix:

[0083]

[0084] Each element of the cost matrix C represents the time step Φ k-1 (That is, the total similarity cost from the first candidate time step to time step i) (i.e., the added first candidate time step).

[0085] Furthermore, path information associated with the candidate time step can be determined based on cost information. Specifically, the path information can be represented as a path matrix:

[0086]

[0087] After constructing the path matrix P, the optimal path can be traced back based on the path information to determine the multiple time steps to be added to the time step set.

[0088] Therefore, through dynamic programming, embodiments of this disclosure can determine a sharing strategy for conditional features, thereby accelerating inference while reducing the computational cost of the coding unit.

[0089] Example process

[0090] Figure 3 A flowchart of an example process 300 for generating media content according to some embodiments of the present disclosure is shown. Process 300 can be implemented in an electronic device 110.

[0091] As shown in the figure, in box 310, electronic device 110 receives a media generation request.

[0092] In block 320, electronic device 110 uses the coding unit to determine the conditional features corresponding to the first noise reduction time step based on the media generation request and the first noise representation of the first noise reduction time step.

[0093] In block 330, electronic device 110 uses a decoding unit to determine a second noise representation for a second noise reduction time step based on a first noise representation and conditional features.

[0094] In box 340, electronic device 110 generates media content for a media generation request based on a second noise representation.

[0095] In some embodiments, determining the conditional features corresponding to the first noise reduction time step using the encoding unit based on the media generation request and the first noise reduction time step includes: determining classification information associated with the media content to be generated based on the media generation request; and providing the encoding unit with the first noise representation, the first noise reduction time step, and the classification information to generate the conditional features.

[0096] In this way, by introducing classification information associated with media content, the model can more accurately understand the intent behind the request, thereby generating content that better meets user needs. Classification information provides additional context for the generation of conditional features, making the generated features more semantically relevant and accurate, thus improving the quality of the generated content.

[0097] In some embodiments, determining the second noise representation of the second noise reduction time step using the decoding unit based on the first noise representation and conditional features includes: providing the decoding unit with the first noise representation, the first noise reduction time step, and conditional features to determine the velocity features corresponding to the first noise reduction time step; and determining the second noise representation of the second noise reduction time step based on the first noise representation and the velocity features.

[0098] In this way, the embodiments of this disclosure enable the model to dynamically adjust the detail restoration during the generation process, especially in the restoration of high-frequency details, thereby further improving the quality of the generated content.

[0099] In some embodiments, determining the conditional features corresponding to the first noise reduction time step using the encoding unit based on the media generation request and the first noise representation of the first noise reduction time step includes: in response to the first noise reduction time step matching a preset time step set, determining the conditional features corresponding to the first noise reduction time step using the encoding unit based on the media generation request and the first noise representation.

[0100] In this way, by using a pre-defined set of time steps, the model recalculates conditional features at specific time steps while sharing them at other time steps. This method leverages the local consistency of conditional features between adjacent time steps, reducing unnecessary computation and improving inference efficiency. Furthermore, by dynamically adjusting the recalculation time steps of conditional features, the model can better balance computational resources and generation quality.

[0101] In some embodiments, the condition feature is a first condition feature, and generating media content for a media generation request based on a second noise representation includes: in response to a mismatch between a second noise reduction time step and a preset set of time steps, determining a second condition feature corresponding to the second noise reduction time step based on the first condition feature; using a decoding unit to determine a third noise representation corresponding to a third noise reduction time step based on the second condition feature and the second noise representation; and generating media content based on the third noise representation.

[0102] In this way, when the second noise reduction time step is not in the preset time step set, the model dynamically generates the second condition feature based on the existing first condition feature, thereby reducing the computational cost of the coding unit.

[0103] In some embodiments, determining the second condition feature corresponding to the second noise reduction time step based on the first condition feature includes: determining the time difference between the first noise reduction time step and the second noise reduction time step; and determining the second condition feature based on the first condition feature and the time difference.

[0104] In this way, by introducing a time difference, the embodiments of this disclosure more precisely control the sharing of conditional features, thereby further improving inference efficiency.

[0105] In some embodiments, a preset set of time steps is constructed based on the following process: processing a set of sample data using an encoding unit and a decoding unit to determine similarity information associated with multiple denoising time steps, the similarity information indicating the similarity of conditional features corresponding to different denoising time steps; and determining a preset set of time steps from the multiple denoising time steps based on the similarity information.

[0106] In this way, by constructing a preset set of time steps based on similarity information, embodiments of this disclosure can select the time steps where conditional features need to be recalculated. This data-driven method can dynamically adjust the set of time steps according to the actual feature similarity, thereby achieving a better balance between performance and efficiency on different datasets and tasks.

[0107] In some embodiments, determining a preset set of time steps from multiple noise reduction time steps based on similarity information includes: determining cost information associated with candidate time steps based on similarity information; constructing path information associated with candidate time steps based on cost information; and determining a preset set of time steps from multiple noise reduction time steps based on path information.

[0108] In this way, by using a dynamic programming algorithm to select the optimal time step from candidate time steps and add it to a preset time step set, the model can ensure that the selected time step set is globally optimal. This method not only improves the efficiency of time step set selection but also further optimizes the model's inference performance, enabling it to generate high-quality media content more efficiently in practical applications.

[0109] In some embodiments, the decoding unit is configured to dynamically determine normalization parameters based on conditional features to determine a second noise representation.

[0110] In this way, by dynamically adjusting the normalization parameters, the decoding unit can better adapt to feature changes at different time steps, enhancing the model's flexibility and adaptability. This dynamic normalization mechanism enables the model to more stably recover high-frequency details during the generation process, further improving the quality of the generated content.

[0111] In some embodiments, the encoding unit is trained based on the following process: acquiring a first intermediate feature determined by the encoding unit in processing a training generation request; mapping the first intermediate feature to a feature space corresponding to a reference feature to determine a second intermediate feature, the reference feature being generated based on training media content corresponding to the training generation request; and training the encoding unit based on the difference between the second intermediate feature and the reference feature.

[0112] In this way, by aligning the intermediate features generated by the encoding units with reference features during training, the model can significantly improve its ability to extract conditional features, making the generated conditional features more semantically consistent and stable. This training method helps reduce feature redundancy, while accelerating model training convergence and improving generation efficiency.

[0113] Example devices and equipment

[0114] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4A schematic structural block diagram of an example apparatus 400 for generating media content according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in electronic device 110. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0115] like Figure 4 As shown, the apparatus 400 includes a receiving module 410 configured to receive a media generation request; an encoding module 420 configured to use an encoding unit to determine a conditional feature corresponding to a first noise reduction time step based on the media generation request and a first noise representation of a first noise reduction time step; a decoding module 430 configured to use a decoding unit to determine a second noise representation of a second noise reduction time step based on the first noise representation and the conditional feature; and a generation module 440 configured to generate media content for the media generation request based on the second noise representation.

[0116] In some embodiments, the encoding module 420 is further configured to: determine classification information associated with the media content to be generated based on the media generation request; and provide the encoding unit with a first noise representation, a first noise reduction time step, and classification information to generate conditional features.

[0117] In this way, by introducing classification information associated with media content, the model can more accurately understand the intent behind the request, thereby generating content that better meets user needs. Classification information provides additional context for the generation of conditional features, making the generated features more semantically relevant and accurate, thus improving the quality of the generated content.

[0118] In some embodiments, the decoding module 430 is further configured to: provide the decoding unit with a first noise representation, a first noise reduction time step, and conditional features to determine a speed feature corresponding to the first noise reduction time step; and determine a second noise representation for the second noise reduction time step based on the first noise representation and the speed feature.

[0119] In this way, the embodiments of this disclosure enable the model to dynamically adjust the detail restoration during the generation process, especially in the restoration of high-frequency details, thereby further improving the quality of the generated content.

[0120] In some embodiments, the encoding module 420 is further configured to: in response to a first noise reduction time step matching a preset set of time steps, utilize the encoding unit to determine a conditional feature corresponding to the first noise reduction time step based on a media generation request and a first noise representation.

[0121] In this way, by using a pre-defined set of time steps, the model recalculates conditional features at specific time steps while sharing them at other time steps. This method leverages the local consistency of conditional features between adjacent time steps, reducing unnecessary computation and improving inference efficiency. Furthermore, by dynamically adjusting the recalculation time steps of conditional features, the model can better balance computational resources and generation quality.

[0122] In some embodiments, the condition feature is a first condition feature, and the generation module 440 is further configured to: in response to a mismatch between the second noise reduction time step and a preset time step set, determine a second condition feature corresponding to the second noise reduction time step based on the first condition feature; use the decoding unit to determine a third noise representation corresponding to the third noise reduction time step based on the second condition feature of the second noise representation; and generate media content based on the third noise representation.

[0123] In this way, when the second noise reduction time step is not in the preset time step set, the model dynamically generates the second condition feature based on the existing first condition feature, thereby reducing the computational cost of the coding unit.

[0124] In some embodiments, the generation module 440 is further configured to: determine the time difference between the first noise reduction time step and the second noise reduction time step; and determine the second condition feature based on the first condition feature and the time difference.

[0125] In this way, by introducing a time difference, the embodiments of this disclosure more precisely control the sharing of conditional features, thereby further improving inference efficiency.

[0126] In some embodiments, a preset set of time steps is constructed based on the following process: processing a set of sample data using an encoding unit and a decoding unit to determine similarity information associated with multiple denoising time steps, the similarity information indicating the similarity of conditional features corresponding to different denoising time steps; and determining a preset set of time steps from the multiple denoising time steps based on the similarity information.

[0127] In this way, by constructing a preset set of time steps based on similarity information, embodiments of this disclosure can select the time steps where conditional features need to be recalculated. This data-driven method can dynamically adjust the set of time steps according to the actual feature similarity, thereby achieving a better balance between performance and efficiency on different datasets and tasks.

[0128] In some embodiments, determining a preset set of time steps from multiple noise reduction time steps based on similarity information includes: determining cost information associated with candidate time steps based on similarity information; constructing path information associated with candidate time steps based on cost information; and determining a preset set of time steps from multiple noise reduction time steps based on path information.

[0129] In this way, by using a dynamic programming algorithm to select the optimal time step from candidate time steps and add it to a preset time step set, the model can ensure that the selected time step set is globally optimal. This method not only improves the efficiency of time step set selection but also further optimizes the model's inference performance, enabling it to generate high-quality media content more efficiently in practical applications.

[0130] In some embodiments, the decoding unit is configured to dynamically determine normalization parameters based on conditional features to determine a second noise representation.

[0131] In this way, by dynamically adjusting the normalization parameters, the decoding unit can better adapt to feature changes at different time steps, enhancing the model's flexibility and adaptability. This dynamic normalization mechanism enables the model to more stably recover high-frequency details during the generation process, further improving the quality of the generated content.

[0132] In some embodiments, the encoding unit is trained based on the following process: acquiring a first intermediate feature determined by the encoding unit in processing a training generation request; mapping the first intermediate feature to a feature space corresponding to a reference feature to determine a second intermediate feature, the reference feature being generated based on training media content corresponding to the training generation request; and training the encoding unit based on the difference between the second intermediate feature and the reference feature.

[0133] In this way, by aligning the intermediate features generated by the encoding units with reference features during training, the model can significantly improve its ability to extract conditional features, making the generated conditional features more semantically consistent and stable. This training method helps reduce feature redundancy, while accelerating model training convergence and improving generation efficiency.

[0134] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110.

[0135] like Figure 5As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0136] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0137] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0138] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0139] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0140] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0141] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0143] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0145] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating media content, comprising: Receive media generation requests; The coding unit determines the conditional features corresponding to the first noise reduction time step based on the media generation request and the first noise representation of the first noise reduction time step. The decoding unit determines the second noise representation for the second noise reduction time step based on the first noise representation and the conditional features. as well as Based on the second noise representation, media content is generated for the media generation request.

2. The method according to claim 1, wherein determining the conditional features corresponding to the first noise reduction time step using the coding unit based on the media generation request and the first noise representation of the first noise reduction time step includes: Based on the media generation request, determine the classification information associated with the media content to be generated; as well as The first noise representation, the first noise reduction time step, and the classification information are provided to the encoding unit to generate the conditional features.

3. The method according to claim 1, wherein determining the second noise representation of the second noise reduction time step using the decoding unit based on the first noise representation and the conditional features comprises: The decoding unit is provided with the first noise representation, the first noise reduction time step, and the conditional feature to determine the speed feature corresponding to the first noise reduction time step; as well as Based on the first noise representation and the velocity characteristics, the second noise representation of the second noise reduction time step is determined.

4. The method according to claim 1, wherein determining the conditional features corresponding to the first noise reduction time step using the coding unit based on the media generation request and the first noise representation of the first noise reduction time step includes: In response to the first noise reduction time step matching a preset time step set, the encoding unit determines the conditional feature corresponding to the first noise reduction time step based on the media generation request and the first noise representation.

5. The method of claim 4, wherein the conditional feature is a first conditional feature, and generating media content for the media generation request based on the second noise representation comprises: In response to the mismatch between the second noise reduction time step and the preset time step set, a second condition feature corresponding to the second noise reduction time step is determined based on the first condition feature; The decoding unit uses the second noise representation based on the second conditional feature to determine the third noise representation corresponding to the third noise reduction time step; as well as The media content is generated based on the third noise representation.

6. The method according to claim 5, wherein determining the second conditional feature corresponding to the second noise reduction time step based on the first conditional feature includes: Determine the time difference between the first noise reduction time step and the second noise reduction time step; as well as The second condition feature is determined based on the first condition feature and the time difference.

7. The method of claim 4, wherein the preset time step set is constructed based on the following process: The encoding unit and the decoding unit process a set of sample data to determine similarity information associated with multiple denoising time steps, the similarity information indicating the similarity of conditional features corresponding to different denoising time steps; and Based on the similarity information, the preset time step set is determined from the plurality of noise reduction time steps.

8. The method according to claim 7, wherein determining the preset time step set from the plurality of noise reduction time steps based on the similarity information comprises: Based on the similarity information, cost information associated with the candidate time step is determined; Based on the cost information, path information associated with the candidate time step is constructed; as well as Based on the path information, the preset time step set is determined from the plurality of noise reduction time steps.

9. The method of claim 1, wherein the decoding unit is configured to dynamically determine normalization parameters based on the conditional features to determine the second noise representation.

10. The method of claim 1, wherein the coding unit is trained based on the following process: Obtain the first intermediate feature determined by the encoding unit in processing the training generation request; The first intermediate feature is mapped to a feature space corresponding to a reference feature to determine a second intermediate feature, wherein the reference feature is generated based on the training media content corresponding to the training generation request; and The encoding unit is trained based on the difference between the second intermediate feature and the reference feature.

11. An apparatus for generating media content, comprising: The receiving module is configured to receive media generation requests; The encoding module is configured to use the encoding unit to determine the conditional features corresponding to the first noise reduction time step based on the media generation request and the first noise representation of the first noise reduction time step; The decoding module is configured to use the decoding unit to determine the second noise representation of the second noise reduction time step based on the first noise representation and the conditional features; as well as The generation module is configured to generate media content for the media generation request based on the second noise representation.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 10.

Images