Methods, apparatus, devices, media, and programs for generating media content
By reusing directional and feature data from machine learning models to generate media content, the problem of slow inference speed caused by the large number of iterations in existing models is solved, thus achieving efficient media content generation.
Patent Information
- Application Number
- CN202611336553.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing generative models require a large number of iterations in tasks involving the generation of media content such as images and videos, resulting in slow inference speeds and making it difficult to meet the needs of real-time interaction or large-scale deployment scenarios.
By acquiring the directional and feature data obtained from the same processing of the machine learning model, the intermediate representation is processed based on this data to determine the target direction and region, perform state deviation compensation, and use the baseline state and state deviation information to perform a state update to generate media content.
While maintaining the quality of generated content, it reduces redundant model processing and intermediate data transfer, thereby reducing computational overhead and processing time and improving the efficiency of media content generation.
Smart Images

Figure CN123123672A_ABST
Abstract
Description
Technical Field
[0001] Examples of this disclosure relate to the field of computers and to methods, apparatus, devices, media, and program products for generating media content. Background Technology
[0002] Generative models are widely used in tasks involving the generation of media content such as images, videos, and audio. These models typically transform randomly sampled initial states into target media content through iterative steps. To achieve high generation quality, these models often require a large number of iterations, resulting in slow inference speeds and making it difficult to meet the needs of real-time interactive or large-scale deployment scenarios. Summary of the Invention
[0003] In a first aspect of this disclosure, a method for generating media content is provided, comprising: acquiring first direction data and first feature data, wherein the first direction data and the first feature data are obtained by a machine learning model processing the first intermediate representation in the same step, the first intermediate representation being an intermediate state of the media content to be generated; processing the first intermediate representation based on the first direction data and the first feature data to obtain a second intermediate representation; and generating the media content based on the second intermediate representation. This method can reuse the direction data and feature data obtained by the machine learning model in the same step and update the intermediate state of the generation process accordingly, thereby reducing redundant processing and balancing the generation quality and computational overhead of fewer sampling steps.
[0004] In some embodiments, processing the first intermediate representation includes: determining a target direction for processing the first intermediate representation based on the first direction data and reference direction data, wherein the reference direction data is determined based on at least one historical direction data from the machine learning model; and processing the first intermediate representation based on the target direction. Thus, by determining the reference direction using historical direction data, the compensation direction is aligned with the inertial direction of the currently generated trajectory, thereby improving the accuracy of the compensation.
[0005] In some embodiments, the reference direction data is determined by: determining the weight of at least one historical direction data point, wherein the weight of the historical direction data point is inversely correlated with the time step of the historical direction data point relative to the first direction data point; and determining the reference direction data based on the at least one historical direction data point and the determined weights. Thus, historical direction data closer to the current time step contributes more to the reference direction, enabling the reference direction to more accurately reflect the inertia of recent trajectories.
[0006] In some embodiments, processing the first intermediate representation includes: determining a target region for processing the first intermediate representation based on the first feature data; and processing the first intermediate representation based on the target region. Determining the target region for processing using feature data allows the processing to focus on spatial locations in the generated content where details are easily lost.
[0007] In some embodiments, determining the target region includes: dividing the feature data into multiple sets of regional feature data, the multiple sets of regional feature data corresponding to multiple regions of the media content; determining a change measure of the multiple sets of regional feature data in a preset dimension; and determining the target region based on the corresponding change measure of the multiple sets of regional feature data. Locating the spatial region to be processed by the change measure of the regional feature data avoids ineffective processing of stable regions.
[0008] In some embodiments, the change measure includes at least one of the following: a first change measure in the spatial dimension, the first change measure being determined based on feature values at multiple spatial locations within one of the plurality of regions; a second change measure in the temporal dimension, the second change measure being determined based on feature values of the region across multiple time frames; and a third change measure in the channel dimension, the third change measure being determined based on feature values of the region across multiple channels. Thus, the statistics of the change measure can adapt to the feature structures of various media content such as images and videos, enhancing the versatility of the solution.
[0009] In some embodiments, the method further includes at least one of the following: performing a size adjustment on the location data corresponding to the target region so that the location data matches the first intermediate representation in spatial dimension; and performing channel mapping on the location data corresponding to the target region so that the location data matches the first intermediate representation in the number of channels. In this way, the location data can be aligned with the intermediate representation in both spatial and channel dimensions, ensuring that processing actions can be accurately applied to the corresponding positions in the intermediate representation.
[0010] In some embodiments, processing the first intermediate representation based on the first direction data and the first feature data includes: determining state deviation information based on the first direction data and the first feature data; and processing the first intermediate representation based on the state deviation information to obtain the second intermediate representation. The state deviation information is jointly generated by the direction data and feature data, and possesses both directional guidance and spatial selectivity, thus avoiding indiscriminate disturbances to the stable region.
[0011] In some embodiments, processing the first intermediate representation based on the state deviation information includes: determining a reference state based on the first intermediate representation and the first direction data; and processing the first intermediate representation based on the reference state and the state deviation information to obtain the second intermediate representation. By introducing a reference state, the processing maintains the original main generation path of the model while using the state deviation information to enhance local details in a targeted manner.
[0012] In some embodiments, processing the first intermediate representation based on the baseline state and the state deviation information to obtain the second intermediate representation includes: determining state update data based on the constraint strength controlling the regression of the first intermediate representation to the baseline state, the state deviation information, and the baseline state, wherein the state update data indicates the update center position from the first intermediate representation to the second intermediate representation; determining random offset data based on the constraint strength and the random offset strength, wherein the random offset data indicates the random offset range from the first intermediate representation to the second intermediate representation; and performing a state update on the first intermediate representation based on the state update data and the random offset data to obtain the second intermediate representation. Thus, the processing can be completed with a single state update, eliminating the need for additional iterative steps and avoiding an increase in the number of forward calls to the model; simultaneously, the constraint strength can limit the deviation caused by the state deviation information, reducing the risk of distortion in the generated content.
[0013] In some embodiments, the method further includes adjusting the constraint strength based on at least one of sampling time, the target region, the magnitude of the difference between the first direction data and the reference direction data, and the number of sampling steps. Thus, the constraint strength can be dynamically adjusted according to the characteristics of the sampling progress or the current state, allowing for a larger exploration space to recover details in the early sampling phase and enhancing stability to reduce artifacts in the later sampling phase.
[0014] In some embodiments, the parameters of the machine learning model remain unchanged during the generation of the media content, and / or the first direction data and the first feature data are obtained by the same processing performed by the machine learning model on the first intermediate representation. Thus, without modifying or retraining the existing model, or additionally calling the model to obtain feature data, the generation quality of fewer sampling steps can be improved without increasing inference latency.
[0015] In some embodiments, the machine learning model includes at least one of a diffusion model, a flow matching model, and a rectified flow model. This allows for compatibility with multiple mainstream generative model architectures and provides broad applicability.
[0016] In a second aspect of this disclosure, an apparatus for generating media content is provided, comprising: an acquisition module configured to acquire first direction data and first feature data, wherein the first direction data and the first feature data are obtained by a machine learning model processing the first intermediate representation in the same step, the first intermediate representation being an intermediate state of the media content to be generated; a processing module configured to process the first intermediate representation based on the first direction data and the first feature data to obtain a second intermediate representation; and a generation module configured to generate the media content based on the second intermediate representation. This apparatus can reuse the direction data and feature data obtained by the machine learning model in the same step and update the intermediate state of the generation process accordingly, thereby reducing redundant processing and balancing the generation quality and computational overhead of fewer sampling steps.
[0017] In some embodiments, the processing module is further configured to: determine a target direction for processing the first intermediate representation based on the first direction data and reference direction data, wherein the reference direction data is determined based on at least one historical direction data of the machine learning model; and process the first intermediate representation based on the target direction. Thus, by determining the reference direction using historical direction data, the compensation direction is aligned with the inertial direction of the currently generated trajectory, thereby improving the accuracy of the compensation.
[0018] In some embodiments, the processing module is further configured to: determine the weight of at least one historical direction data, wherein the weight of the historical direction data is inversely correlated with the time step of the historical direction data relative to the first direction data; and determine the reference direction data based on the at least one historical direction data and the determined weight. Thus, the historical direction data closer to the current time step contributes more to the reference direction, enabling the reference direction to more accurately reflect the inertia of recent trajectories.
[0019] In some embodiments, the processing module is further configured to: determine a target region for processing the first intermediate representation based on the first feature data; and process the first intermediate representation based on the target region. Thus, by determining the target region to be processed using feature data, the processing actions are focused on spatial locations in the generated content where details are easily lost.
[0020] In a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform any of the methods of the first aspect described above when executed by the at least one processor.
[0021] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executable by a processor to implement the method of any of the first aspects described above.
[0022] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of any one of the first aspects described above.
[0023] In this way, by reusing the directional and feature data obtained from the same processing of the machine learning model to process the intermediate state of media content, it is possible to reduce repeated model processing and intermediate data transmission while maintaining the quality of generation, thereby reducing the computational overhead and processing time of low-step inference and improving the generation efficiency of media content.
[0024] It should be understood that the content described in this content section is not intended to limit the key or essential features of the examples in this disclosure, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description
[0025] The above and other features, advantages, and aspects of the various examples in this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 An example system architecture for media generation model inference based on some embodiments is shown; Figure 3 A flowchart is shown as an example process of low-step inference for a media generation model according to some embodiments; Figure 4 A flowchart is shown illustrating an example method for generating media content according to some embodiments; Figure 5 A schematic structural block diagram of an example apparatus for generating media content according to some embodiments is shown; and Figure 6 A block diagram is shown in which one or more examples of an electronic device can be implemented. Detailed Implementation
[0026] Examples of this disclosure will now be described in more detail with reference to the accompanying drawings. While some examples of this disclosure are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples set forth herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.
[0027] It should be noted that the headings of any section / subsection provided in this disclosure are not limiting. This disclosure describes a variety of examples throughout, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.
[0028] In the description of examples in this disclosure, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0029] The examples in this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and provisions. In the examples of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.
[0030] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.
[0031] As mentioned above, to achieve high generation quality, these models often require a large number of iterations, resulting in slow inference speeds and making it difficult to meet the needs of real-time interaction or large-scale deployment scenarios. To reduce inference latency, related technologies attempt to reduce the number of iterations. However, with fewer sampling steps, the model struggles to fully recover the structure and details of the target content within a limited number of steps, leading to decreased fidelity and structural instability in the generated results.
[0032] Few-step deterministic sampling in the form of Ordinary Differential Equations (ODEs) can compress multimodal local variations to near the conditional mean. This compression flattens textures, edges, and local structures. Correction in the form of Stochastic Differential Equations (SDEs) can introduce stochasticity. If related paths depend on multiple stochastic microsteps, this can increase the number of model calls and discrete truncation errors.
[0033] Some training-phase drift paths learn distribution evolution through new training objectives. Other autoregressive paths leverage prediction uncertainty or specific drift fields to accelerate decoding. Still others are adaptive paths that learn sampling scheduling by training a policy network. The inference path here does not train a new drift model, nor does it rely on prediction entropy or a policy network. This inference path utilizes the frozen model's own output history and intermediate features to construct an anchor-constrained analytic state update within each sampling step.
[0034] This disclosure proposes a scheme for generating media content. The scheme acquires first-direction data and first-feature data obtained by a machine learning model processing a first intermediate representation in the same step. Based on the first-direction data and first-feature data, it processes the first intermediate representation to obtain a second intermediate representation. The machine learning model then generates media content based on this second intermediate representation. This approach reduces repetitive model processing and intermediate data transfer while maintaining generation quality, thereby reducing the computational overhead and processing time of low-step inference and improving the efficiency of media content generation.
[0035] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.
[0036] Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, environment 100 includes electronic device 110, application 120, server 130, user 140, and interface 150.
[0037] Electronic device 110 runs application 120 that supports media content generation. User 140 can interact with application 120 through electronic device 110, and electronic device 110 presents interface 150 through application 120.
[0038] Electronic device 110 communicates with server 130 to obtain background processing from application 120. The bidirectional connection between server 130 and electronic device 110 can transmit conditional inputs, sampling status, or generation results.
[0039] Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality (VR) / augmented reality (AR) devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).
[0040] Server 130 can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 in electronic devices 110 that support media content generation.
[0041] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some embodiments, server 130 and electronic device 110 can interact via this communication connection.
[0042] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.
[0043] The examples disclosed herein can be implemented on electronic device 110. It should be noted that the operations described relative to electronic device 110 can be performed by a related application 120 running on electronic device 110. The interface described relative to electronic device 110 can be provided by application 120; for example, electronic device 110 presents a corresponding interface by running application 120. Furthermore, some operations described relative to electronic device 110 may require the assistance of server 130 to complete.
[0044] Environment 100 supports deployment methods such as cloud computing devices, edge electronic devices, workstations, and mobile electronic devices. Inference processing can be performed entirely by server 130 or entirely by electronic device 110. Electronic device 110 can also collaborate with server 130 to perform conditional encoding, model forward processing, state compensation, and result decoding.
[0045] Inputs may include text conditions, image conditions, video conditions, category information, or other control conditions. Outputs may include images, videos, audio, 3D content, scientific simulation samples, or other media content.
[0046] In the few-step inference process, the electronic device 110 iteratively processes the initial sampled state using a machine learning model to ultimately obtain the sampled state used to generate media content. The machine learning model used in this disclosure refers to a parameterized computational model that generates an output field based on the sampled state, sampling time, and conditional input. In the discrete sampling step, the relationship between the current state and the output field is as follows:
[0047] in, This indicates the sampling state of the i-th sampling step. y represents the corresponding continuous time, and y represents the conditional input. Represents a machine learning model. This represents the output field of the model.
[0048] Machine learning models can include diffusion models, stream matching models, rectified stream models, or latent space generation models. Diffusion models obtain media content by progressively changing sample states and performing inverse generation. Stream matching models learn the velocity field between consecutive states. Rectified stream models can be used as an implementation of stream matching models. Optimal transmission refers to a class of methods for constructing state transmission relationships under given distribution constraints; this term can be used to describe the origin of certain flow paths but does not limit the model to a specific transmission construction. Machine learning models can use transformer structures, encoder-decoder structures, or network structures with spatiotemporal processing capabilities. The velocity field can also be obtained from noise prediction, data prediction, velocity prediction, or fractional functions through equivalent transformations.
[0049] The machine learning model disclosed herein is a parameterized neural network model for generating media content. Its inputs are sampling states, sampling time, and conditional inputs, and its output is an output field, ultimately producing user-perceptible media content. The output field has different physical meanings in different model architectures. For example, in a flow matching model, the output field is a velocity field, representing the instantaneous rate of change of the sampling states along the time direction in the state space. In a diffusion model, the output field is a noise prediction or fractional function, representing the gradient direction of the sampling states relative to the target data distribution. In a rectified flow model, the output field is a transport field, representing the displacement direction of the sampling states along the optimal transmission path. These output fields are all continuous vector fields with the same dimension as the sampling states, used to indicate the update direction of the sampling states in the state space.
[0050] The sampling state is an intermediate latent variable in the generation process, and its relationship with the final media content is jointly defined by the machine learning model and its decoder. In the latent space diffusion model, the sampling state is the latent space feature vector output by the encoder, which the decoder maps back to pixel space to generate image or video frames. In the pixel space diffusion model, the sampling state directly corresponds to the pixel values of the noisy image or video frame. Each dimension of the sampling state corresponds to the spatial location, color channel, or temporal frame of the final media content. Conditional inputs are real-world user control signals, including text descriptions, reference images, reference videos, category labels, or other control conditions. These conditions are provided by the user through the application interface to specify the semantic content and stylistic attributes of the media content to be generated.
[0051] The first directional data and the first feature data are outputs obtained by the model performing a forward propagation calculation on the current sampling state, both originating from the same calculation process. The first directional data may include a first output field, which is a directional signal used to update the sampling state. The first feature data may include an intermediate feature map, which is a spatial feature tensor generated by an intermediate layer within the model. Its spatial dimension corresponds to the spatial layout of the final media content. Each spatial location in the feature map corresponds to the local visual information of the corresponding spatial location in the final image or video frame. Therefore, the local change measure of the intermediate feature map reflects the structural complexity and detail richness of that location in the final media content. The target region can be indicated by a spatial mask. Based on this correspondence, the spatial mask can locate regions in the final media content that are rich in detail and prone to losing texture and structure by analyzing the local change measure of the intermediate feature map.
[0052] The following describes the specific implementation of the above reasoning process.
[0053] It should be understood that, although the embodiments are described using electronic device 110 as the execution subject in the examples of this disclosure, this is merely illustrative and does not constitute a limitation on the scope of implementation of the solutions disclosed herein. The processes described in the following embodiments can be executed entirely locally on electronic device 110, entirely on server 130, or jointly by electronic device 110 and server 130.
[0054] First, the electronic device 110 acquires first directional data and first feature data. The first directional data and first feature data are obtained by performing the same processing on a first intermediate representation using a machine learning model. The first intermediate representation is an intermediate state in the media content generation process; it can be a latent variable, a noise state, or other intermediate generated state, located by both discrete sampling steps and continuous sampling time. The first directional data may include a first output field, which can be a velocity field, flow field, noise prediction, or equivalent update direction obtained by the machine learning model performing forward processing on the first intermediate representation. The first feature data may include an intermediate feature map, which can refer to spatial or spatiotemporal features generated by one or more intermediate layers in the forward processing of the same model. The first directional data and first feature data can come from the same forward processing of the model to reduce computational overhead. The machine learning model uses the processed second intermediate representation to continue generating media content, and the final state is converted into a target sample by the output decoding component.
[0055] After obtaining the first direction data, the electronic device 110 needs to determine the target direction for processing the first intermediate representation. The first direction data may include a first output field, the reference direction data may include output field momentum, and the target direction may include the kinematic residual determined by the difference between the first output field and the output field momentum. The output field momentum is a historical summation based on at least one historical output field weighted accordingly, representing the inertial direction of the sampled trajectory.
[0056] Electronic device 110 determines the weights of historical directional data. Historical directional data may include historical output fields. The weight of a historical directional data point is inversely correlated with its time step relative to the first directional data point; that is, historical directional data points closer to the current sampling step are assigned a larger weight, and those farther away have a smaller weight. Historical directional data can be further restricted to the output fields corresponding to a preset number of sampling steps prior to the first intermediate representation, to control computational overhead. Based on these historical directional data points and their weights, electronic device 110 determines reference directional data. For example, the reference directional data may be the output field momentum. Through this weighting method, the reference directional data can reflect the inertial direction of recent sampling trajectories, reducing the interference of instantaneous direction changes on subsequent target directions.
[0057] Based on the difference between the first output field and the output field momentum, the electronic device 110 determines the kinematic residual. The kinematic residual indicates the direction of compensation for the first intermediate representation and is aligned with the current sampling trajectory itself. For example, the kinematic residual can be obtained by subtracting the previous output field momentum from the current output field before momentum update, or it can be obtained by updating the output field momentum first and then subtracting the updated output field momentum from the current output field. The two calculation methods are equivalent in direction.
[0058] In some embodiments, if the calculation method prior to momentum update is used:
[0059] in, Represents kinematic residuals, Indicates the output field of the current sampling step. This represents the output field momentum of the previous sampling step.
[0060] In other embodiments, if a calculation method following momentum update is used, then momentum update is performed first, followed by calculation of the residuals:
[0061] in, This represents the updated output field momentum after the current sampling step. Indicates the output field of the current sampling step. This represents the output field momentum of the previous sampling step. The momentum decay coefficient controls the degree to which historical information is retained.
[0062] The two residuals maintain the same direction, and the scale difference can be absorbed by the time decay coefficient. The residuals can also be estimated using adjacent output field differences, multi-order output field differences, or acceleration, as long as the obtained quantities can reflect the direction of the current trajectory relative to historical inertia.
[0063] After obtaining the first direction data and the first feature data, the electronic device 110 also needs to determine the target region for processing the first intermediate representation. The first feature data may include an intermediate feature map, and the target region may be indicated by a spatial mask determined based on the intermediate feature map. The spatial mask indicates the spatial region for processing the first intermediate representation.
[0064] In some embodiments, the electronic device 110 divides the intermediate feature map into multiple regions, determines the variance of each region in at least one dimension, and then determines a spatial mask based on the corresponding variance of each region. The first variance in the spatial dimension is determined based on feature values of multiple spatial locations within a region. The second variance in the temporal dimension is determined based on feature values of a region across multiple time frames. The third variance in the channel dimension is determined based on feature values of a region across multiple channels. In this way, the spatial mask can locate regions with high local variance in the intermediate feature map. These regions correspond to locations in the generated content that are rich in detail and prone to losing detail in few sampling steps, thereby focusing compensation actions on these regions and avoiding ineffective perturbation of flat regions.
[0065] In one specific implementation, the electronic device 110 calculates the mean and variance of each channel of the intermediate feature map, and performs normalization on each channel. A stabilization term is used to avoid the normalization denominator being zero. For example, for the intermediate feature map... The spatial mean and variance of its c-th channel are calculated as follows:
[0066]
[0067] Where H and W represent the height and width of the feature map, respectively. Let represent the feature value of the i-th step, the c-th channel, and the position (h, w). This is the spatial mean of the channel. This represents the spatial variance of the channel. Normalization is performed on each channel:
[0068] in, These are the normalized eigenvalues. To prevent the stable term from being divided by zero.
[0069] After normalization, electronic device 110 calculates the local variance along the channel dimension and performs normalization on the obtained variance to obtain a spatial mask that reflects the local change requirements:
[0070]
[0071] in, This represents the variance across the channel at position (h, w). This is the normalized space mask.
[0072] For video models, intermediate feature maps can further include a temporal dimension. Electronic device 110 can directly form a spatiotemporal mask, or it can form a spatial mask and a temporal mask separately and then fuse them. When the spatial mask and the first intermediate representation have different spatial dimensions, electronic device 110 matches them in spatial dimensions through upsampling, downsampling, or other size adjustments. When the spatial mask and the first intermediate representation have different channel numbers, electronic device 110 matches them in channel number through channel projection, channel mapping, or broadcasting. Alternatively, the spatial mask can be determined based on the fusion result of attention entropy, cross-frame consistency error, local gradient, output prediction variance, or multiple metrics. The mask can also be cropped, smoothed, or normalized.
[0073] After determining the target direction and target area, the electronic device 110 determines state deviation information based on the first direction data and the first feature data. For example, the target direction may include a kinematic residual, the target area may be indicated by a spatial mask, and the state deviation information may include a residual drift determined based on the kinematic residual and the spatial mask. The residual drift is the actual signal quantity processed from the first intermediate representation. For example:
[0074] in, For residual drift, For kinematic residuals, For space mask, This is the time decay coefficient.
[0075] The time decay coefficient decreases as the sampling process progresses, resulting in a larger residual drift scale during the coarse structure generation stage and a smaller scale near the final sample. The time decay coefficient can take one of the following two forms:
[0076] or
[0077] in, The initial residual drift scale, The attenuation factor is a monotonically decreasing factor that decreases with each sampling step. This refers to the current continuous time.
[0078] Therefore, the spatial mask causes residual drift to mainly affect areas with high local variance, and the time coefficient weakens the compensation intensity in the later stages of sampling.
[0079] To maintain the stability of the generated trajectory during processing and prevent state deviation information from causing the sampled state to deviate from the data manifold, the electronic device 110 determines a reference state based on a first intermediate representation and first direction data. For example, the first direction data may include a first output field, and the reference state may include a drift-free state. The drift-free state refers to the state obtained by updating the first intermediate representation according to the reference method without adding residual drift, and can serve as the local manifold anchor point for the current macroscopic time step. The drift-free state is determined by the following formula:
[0080] in, In a drift-free state, This is the baseline update function, i.e., a one-step deterministic update method for the original sampler. This is the current sampling state. For the current continuous time, The step size of the current sampling step. This is the current output field.
[0081] In one implementation, the baseline update method can be expressed as:
[0082] in, This is the current sampling state. The step size of the current sampling step. The velocity field / output field output by the model. This is a drift-free state (local manifold anchor point).
[0083] This approach utilizes the one-step deterministic update result of the original sampler. By introducing a drift-free state as a baseline, the compensation process can enhance local details in a targeted manner by leveraging residual drift while maintaining the original master generation path of the model.
[0084] Based on the baseline state and state deviation information, electronic device 110 processes the first intermediate representation through a single state update to obtain the second intermediate representation. For example, the baseline state may include a drift-free state, and the state deviation information may include residual drift. This single state update can be implemented through an analytical mean-recovery process. This process has mean-recovery properties and can directly complete the state update using the closed-loop transition mean and transition covariance, without requiring multiple iterative microsteps. Within each macroscopic time step, electronic device 110 establishes the following stochastic process around the drift-free state:
[0085] in, For local micro-time, For state variables in local infinitesimal time, To restore stiffness, the restoring stiffness can be used as a constraint strength and control the state towards a drift-free state. The strength of the response, For residual drift, The random buffer strength can be used as the random offset strength. This is a standard Wiener process (i.e., Brownian motion). For its increment, This represents a local micro-time increment.
[0086] This stochastic process has a closed-loop solution. Electronic device 110 determines the transition mean based on recovery stiffness, residual drift, and the drift-free state. The transition mean can be used as state update data and indicates the update center location.
[0087] in, For the transition mean, This is a drift-free state (local manifold anchor point). To restore stiffness, The step size of the current sampling step. This is the current sampling state. This refers to residual drift.
[0088] Meanwhile, the electronic device 110 determines the transfer covariance based on the recovery stiffness and random buffer strength. The transfer covariance can be used as random offset data and indicate the range of random offsets.
[0089] in, To transfer the covariance matrix, For random buffer strength, To restore stiffness, The step size of the current sampling step. It is an identity matrix.
[0090] Subsequently, electronic device 110 samples random quantities from a standard normal distribution. Perform a state jump according to the following formula to obtain the second intermediate representation:
[0091] in, For the transition mean, For random buffer strength, To restore stiffness, The step size of the current sampling step. For a standard normally distributed random quantity , It is a natural exponential function.
[0092] The recovery stiffness can be set to a constant, or it can be adaptively adjusted based on at least one of the following: sampling time, spatial mask, amplitude of kinematic residuals, number of sampling steps, or quality assessment results. As can be seen from the closed-form solution, the expected offset caused by residual drift is limited by the recovery stiffness.
[0093] in, The transition mean (including compensation) is used to introduce residual drift. This is the mean recovery path around the same anchor point without residual drift. The modulus representing the expected offset caused by residual drift (i.e. and (distance between) The residual drift vector, The modulus of residual drift. To restore stiffness, The step size of the current sampling step. It is a natural exponential function.
[0094] The covariance of a random buffer also has a corresponding upper bound:
[0095] in, The transition covariance matrix (i.e., the covariance matrix of random shifts during a single state jump). Let κ be the random buffer strength and κ be the recovery stiffness. Let I be the step size of the current sampling step, and let I be the identity matrix.
[0096] Therefore, the recovery stiffness simultaneously constrains the expected offset caused by residual drift and the covariance of the stochastic buffer. A larger recovery stiffness results in a smaller range of compensated deviations and stochastic offsets. A single state jump provides both directional compensation and bounded stochastic offsets, while closed-loop updates avoid multiple iterative microsteps and additional model calls.
[0097] In this disclosure, the parameters of the machine learning model remain unchanged during media content generation. The first output field and intermediate feature maps are obtained from the same forward processing of the model, and the inference component directly reuses the results of this processing. The forward processing is performed only once per sampling step, therefore state compensation does not increase the number of Neural Function Evaluation (NFE) iterations. Model inference only requires exposing the output field interface and the intermediate internal state interface. When the intermediate feature map cannot be directly output, latent variable local statistics, attention features, residual block activation, or output field local variance can be used as alternatives. The inference path does not require modification of the base model training process. Auxiliary constraints can be set during the training phase to make intermediate features more suitable for representing local changes, but this is not a necessary condition for implementing the inference method. Through the above methods, parameter freezing and reuse of the same forward processing avoid retraining and repeated model computation, reducing inference latency, computational resource consumption, and energy consumption. Simultaneously, the same compensation component can reuse a unified state update framework, adapting to multiple model types and reducing the overhead of repeatedly implementing and maintaining computational kernels for different models.
[0098] like Figure 2 As shown, business request 210 provides text conditions, image conditions, video conditions, or other control conditions. These conditions are sent to a short-step time scheduler 220, which sets the sampling time series and sampling steps, and, based on the computing resources and scheduling status provided by the execution environment 270, hands over the current sampling state to the frozen base model 230. The execution environment 270 may include cloud GPUs, edge computing devices, or chips to provide computational support for the inference process.
[0099] The frozen base model 230 performs a forward processing on the current sampled state, outputting a first output field and an intermediate feature map, which are then provided to the self-aligned drift inference component 240. The self-aligned drift inference component 240 determines the compensation direction based on the momentum residual, determines the compensation region based on the variance mask, and performs state updates through analytical state jumps (such as Ornstein-Uhlenbeck (OU) jumps) to obtain subsequent sampled states. Throughout the inference process, the model forward processing is performed only once per sampling step, requiring no additional training or additional steps.
[0100] The state formed by component 240 is used to obtain the generated result 250. The generated result 250 can be an image, video, audio, 3D content, or other media content. The corresponding sampling state is recorded as a sampling state trajectory 260, which is fed back to the frozen base model 230 for use in the next sampling step, thereby forming a sampling loop.
[0101] The optional quality assessment component 280 can generate assessment results based on factors such as generated quality distance (FID), consistency index, inference latency, or memory usage. These results can be used to select or adjust inference parameters (such as recovery stiffness, stochastic buffer strength, etc.) or to record the running status, but do not require increasing the number of forward passes of the model.
[0102] In this way, the system 200 can perform targeted and regional compensation of the sampling state by using the model's own output history and intermediate features, while keeping the base model parameters unchanged, thus balancing generation quality and inference efficiency under the condition of fewer sampling steps.
[0103] Figure 3 An example process 300 for low-step inference based on a media generation model under certain conditions is shown. Process 300 can be implemented at electronic device 110 or server 130. See below for reference. Figure 3 Describe the process 300.
[0104] In S301, the electronic device 110 initializes the sampling state from a prior distribution (such as a standard Gaussian distribution) and initializes the output field momentum buffer (e.g., sets the momentum to zero). This initialization operation ensures that the inference process has a consistent starting point.
[0105] In S302, the electronic device 110 invokes the machine learning model once to perform forward processing on the current sampled state, obtaining the velocity field (i.e., the output field) and intermediate feature maps. The model forward processing is performed only once per sampling step to control computational overhead.
[0106] In S303, electronic device 110 extracts kinematic residuals based on momentum history. Specifically, electronic device 110 determines the kinematic residuals based on the difference between the current velocity field and the output field momentum, which indicates the direction for compensation of the current sampled state. The output field momentum is determined based on at least one historical output field and can be maintained using an exponential moving average.
[0107] In S304, the electronic device 110 normalizes the intermediate feature map and calculates a spatial variance mask. This mask indicates the spatial region for compensation of the current sampling state, so that subsequent compensation actions are focused on locations with high local variance and where details are easily lost.
[0108] In S305, the electronic device 110 generates a residual drift that is self-aligned with the current trajectory and enhances the corresponding region according to a spatial mask. The residual drift is obtained by modulating the kinematic residual through a spatial mask and an optional time coefficient, which decays as the sampling process progresses, thus weakening the compensation intensity in the later stages of sampling.
[0109] In S306, the electronic device 110 determines the manifold anchor point (i.e., the drift-free state) according to the reference update method and obtains the deterministic predicted position. The drift-free state is the state obtained by the current sampling state according to the reference update method without adding residual drift, and serves as the reference anchor point for the residual drift operation.
[0110] In S307, electronic device 110 calculates the closed-loop transfer mean and transfer covariance using an analytical mean recovery kernel. Specifically, electronic device 110 determines the transfer mean based on recovery stiffness, residual drift, and drift-free state, and determines the transfer covariance based on recovery stiffness and stochastic buffer strength.
[0111] In S308, electronic device 110 performs a one-step analytical state jump, obtaining the state of the next time step based on the transition mean and transition covariance. This jump is completed directly through a closed-form solution, without the need for multiple iterative microsteps, thus not increasing the number of forward calls to the model.
[0112] In S309, electronic device 110 repeats S302 to S308 until the sampling endpoint is reached. The output decoding component converts the final sampled state into a high-fidelity media sample, generating images, videos, audio, 3D content, or other media content.
[0113] Process 300 performs model forward processing only once per time step. Momentum residuals recover local variations, spatial masks locate compensation regions, and parse state jumps limit deviation compensation and absorb surrogate errors. In this way, Process 300 can improve the texture details and structural stability of generated content with fewer sampling steps by performing directional and regional compensation of the sampled state through self-aligned residual drift during inference, while keeping the base model parameters unchanged and the parameters frozen.
[0114] Figure 4 A flowchart of an example process 400 for generating media content based on certain scenarios is shown. Process 400 can be implemented at electronic device 110. See below for reference. Figure 4 To describe process 400.
[0115] like Figure 4 As shown in box 410, electronic device 110 acquires first direction data and first feature data. The first direction data and first feature data are obtained by performing the same processing on a first intermediate representation by a machine learning model, and the first intermediate representation is the intermediate state of the media content to be generated.
[0116] In box 420, electronic device 110 processes the first intermediate representation based on the first direction data and the first feature data to obtain the second intermediate representation.
[0117] In box 430, electronic device 110 generates media content based on a second intermediate representation.
[0118] In this way, the model inference process can reuse the directional and feature data obtained from the same processing of the machine learning model, and use this data to complete intermediate state updates and media content generation. Since the directional and feature data do not need to be obtained separately through different model processing, the electronic device 110 can reduce repeated model processing and intermediate state writing, thereby reducing the computational overhead and processing time of low-step inference.
[0119] In some embodiments, the electronic device 110 may perform the following operations: determining a target direction for processing a first intermediate representation based on first direction data and reference direction data, wherein the reference direction data is determined based on at least one historical direction data from a machine learning model; and processing the first intermediate representation based on the target direction. In this manner, historical direction data is used to determine the reference direction, aligning the processing direction with the inertial direction of the currently generated trajectory, thereby improving processing accuracy.
[0120] In some embodiments, the electronic device 110 can determine the weight of at least one historical direction data, wherein the weight of the historical direction data is inversely correlated with the time step of the historical direction data relative to the first direction data; and determine reference direction data based on the at least one historical direction data and the determined weight. In this way, the historical direction data closer to the current time step contributes more to the reference direction, enabling the reference direction to more accurately reflect the inertia of recent trajectories.
[0121] In some embodiments, the weight of at least one historical direction data point may decay exponentially based on a time step, and / or, the at least one historical direction data point may include historical direction data corresponding to a preset number of sampling steps prior to the first intermediate representation. In this way, exponential decay and a preset historical window give higher weight to recent historical direction data and limit the influence of earlier history on the reference direction data.
[0122] In some embodiments, the electronic device 110 may determine a target region for processing the first intermediate representation based on first feature data; and process the first intermediate representation based on the target region. In this way, by determining the target region to be processed using feature data, the processing action is focused on spatial locations in the generated content where details are easily lost.
[0123] In some embodiments, the electronic device 110 can divide feature data into multiple sets of regional feature data, each set corresponding to multiple regions of the media content; determine the change measure of the multiple sets of regional feature data in a preset dimension; and determine the target region based on the corresponding change measure of the multiple sets of regional feature data. In this way, the spatial region to be processed is located by measuring the change in regional feature data, avoiding ineffective processing of stable regions.
[0124] In some embodiments, the change measure may include at least one of the following: a first change measure in the spatial dimension, which is determined based on feature values at multiple spatial locations within one of multiple regions; a second change measure in the temporal dimension, which is determined based on feature values of a region across multiple time frames; and a third change measure in the channel dimension, which is determined based on feature values of a region across multiple channels. In this way, the statistics of the change measure can adapt to the feature structures of various media content such as images and videos, enhancing the versatility of the solution.
[0125] In some embodiments, the electronic device 110 may perform size adjustment on the location data corresponding to the target area to match the location data with the first intermediate representation in spatial size; and / or perform channel mapping on the location data corresponding to the target area to match the location data with the first intermediate representation in the number of channels. In this way, the adjusted location data can be combined with the first intermediate representation element by element, ensuring that the processing action can be accurately applied to the corresponding position of the intermediate representation.
[0126] In some embodiments, the electronic device 110 can determine state deviation information based on first direction data and first feature data; and process the first intermediate representation based on the state deviation information to obtain a second intermediate representation. In this way, the state deviation information is jointly determined by direction data and feature data, which can reflect the compensation direction and compensation position of the intermediate representation processing.
[0127] In some embodiments, the electronic device 110 may determine a reference state based on a first intermediate representation and first direction data; and process the first intermediate representation based on the reference state and state deviation information to obtain a second intermediate representation. In this way, the reference state provides a reference path for the processing and limits the deviation of the processed state from the main generated trajectory of the model.
[0128] In some embodiments, the electronic device 110 can determine state update data based on the constraint strength, state deviation information, and the baseline state that control the regression of the first intermediate representation to the baseline state. The state update data indicates the update center position from the first intermediate representation to the second intermediate representation. Based on the constraint strength and random offset strength, it can determine random offset data, which indicates the random offset range from the first intermediate representation to the second intermediate representation. Based on the state update data and the random offset data, it performs a single state update on the first intermediate representation to obtain the second intermediate representation. In this way, the process can be completed with a single state update, eliminating the need for additional iterative steps and reducing the number of forward calls to the model.
[0129] In some embodiments, the electronic device 110 can also adjust the constraint strength based on at least one of sampling time, target area, magnitude of state deviation information, and number of sampling steps. In this way, the constraint strength can be adjusted with the sampling stage or processing range, allowing a larger exploration space to recover details in the early sampling stage and enhancing stability in the later sampling stage.
[0130] In some embodiments, the parameters of the machine learning model can remain unchanged during the generation of media content, and / or the first direction data and the first feature data can be obtained by the same processing performed on the first intermediate representation by the machine learning model. In this way, by freezing the model parameters and reusing the results of the same processing, intermediate representation processing can be completed without retraining the model.
[0131] In some embodiments, the machine learning model may include at least one of a diffusion model, a flow matching model, and a rectified flow model. In this way, the same inference framework can be adapted to multiple mainstream generative model architectures, and the unified state update interface can reduce the overhead of repeatedly implementing and maintaining computational kernels across different models.
[0132] A corresponding apparatus for implementing the above methods or processes is also provided.
[0133] Figure 5 A schematic structural block diagram of an example apparatus 500 for generating media content according to some embodiments is shown. Apparatus 500 may be implemented as or included in electronic device 110 or server 130. Various modules or components in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.
[0134] like Figure 5 As shown, the device 500 includes an acquisition module 510, a processing module 520, and a generation module 530. The acquisition module 510 is configured to acquire first direction data and first feature data, which are obtained by performing the same processing on a first intermediate representation using a machine learning model. The first intermediate representation represents an intermediate state of the media content to be generated. The processing module 520 is configured to process the first intermediate representation based on the first direction data and the first feature data to obtain a second intermediate representation. The generation module 530 is configured to generate media content based on the second intermediate representation.
[0135] In some embodiments, the processing module 520 may also be configured to: determine a target direction for processing the first intermediate representation based on first direction data and reference direction data, wherein the reference direction data is determined based on at least one historical direction data of a machine learning model; and process the first intermediate representation based on the target direction.
[0136] In some embodiments, the processing module 520 may also be configured to: determine the weight of at least one historical direction data, wherein the weight of the historical direction data is inversely correlated with the time step of the historical direction data relative to the first direction data; and determine reference direction data based on the at least one historical direction data and the determined weight.
[0137] In some embodiments, the weight of at least one historical direction data may decay exponentially based on the time step, and / or, at least one historical direction data may include historical direction data corresponding to a preset number of sampling steps prior to the first intermediate representation.
[0138] In some embodiments, the processing module 520 may also be configured to: determine a target region for processing the first intermediate representation based on the first feature data; and process the first intermediate representation based on the target region.
[0139] In some embodiments, the processing module 520 may also be configured to: divide the feature data into multiple sets of regional feature data, the multiple sets of regional feature data corresponding to multiple regions of the media content; determine the change measure of the multiple sets of regional feature data in a preset dimension; and determine the target region based on the corresponding change measure of the multiple sets of regional feature data.
[0140] In some embodiments, the change measure may include at least one of the following: a first change measure in the spatial dimension, the first change measure being determined based on feature values at multiple spatial locations within one of multiple regions; a second change measure in the temporal dimension, the second change measure being determined based on feature values of a region across multiple time frames; and a third change measure in the channel dimension, the third change measure being determined based on feature values of a region across multiple channels.
[0141] In some embodiments, the processing module 520 may also be configured to: perform size adjustment on the location data corresponding to the target area so that the location data matches the first intermediate representation in spatial size; and / or perform channel mapping on the location data corresponding to the target area so that the location data matches the first intermediate representation in the number of channels.
[0142] In some embodiments, the processing module 520 may also be configured to: determine state deviation information based on the first direction data and the first feature data; and process the first intermediate representation based on the state deviation information to obtain a second intermediate representation.
[0143] In some embodiments, the processing module 520 may also be configured to: determine a reference state based on a first intermediate representation and first direction data; and process the first intermediate representation based on the reference state and state deviation information to obtain a second intermediate representation.
[0144] In some embodiments, the processing module 520 may further be configured to: determine state update data based on the constraint strength, state deviation information, and reference state that control the regression of the first intermediate representation to the reference state, wherein the state update data indicates the update center position from the first intermediate representation to the second intermediate representation; determine random offset data based on the constraint strength and random offset strength, wherein the random offset data indicates the random offset range from the first intermediate representation to the second intermediate representation; and perform a state update on the first intermediate representation based on the state update data and the random offset data to obtain the second intermediate representation.
[0145] In some embodiments, the processing module 520 may also be configured to adjust the constraint strength based on at least one of the sampling time, target area, magnitude of state deviation information, and number of sampling steps.
[0146] In some embodiments, the parameters of the machine learning model may remain unchanged during the generation of media content, and / or the first direction data and the first feature data may be obtained by the same processing performed by the machine learning model on the first intermediate representation.
[0147] In some embodiments, the machine learning model may include at least one of a diffusion model, a flow matching model, and a rectified flow model.
[0148] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.
[0149] Figure 6 A block diagram of an electronic device 600 in which one or more examples may be implemented is shown. It should be understood that... Figure 6The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described in this disclosure. Figure 6 The illustrated electronic device 600 can be used to implement the model inference component in the electronic device 110 or server 130 discussed above.
[0150] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.
[0151] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.
[0152] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various examples.
[0153] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.
[0154] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).
[0155] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.
[0156] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0157] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0158] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0159] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0160] Various examples have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used in this disclosure is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.
Claims
1. A method for generating media content, comprising: First direction data and first feature data are obtained. The first direction data and the first feature data are obtained by the machine learning model processing the first intermediate representation in the same way. The first intermediate representation is the intermediate state of the media content to be generated. Based on the first direction data and the first feature data, the first intermediate representation is processed to obtain the second intermediate representation; as well as The media content is generated based on the second intermediate representation.
2. The method of claim 1, wherein processing the first intermediate representation comprises: Based on the first direction data and the reference direction data, a target direction for processing the first intermediate representation is determined, wherein the reference direction data is determined based on at least one historical direction data of the machine learning model; and Based on the target direction, the first intermediate representation is processed.
3. The method of claim 2, wherein the reference direction data is determined based on the following: Determine the weights of at least one historical direction data point, wherein the weight of the historical direction data point is inversely correlated with the time step of the historical direction data point relative to the first direction data point; and The reference direction data is determined based on the at least one historical direction data and the determined weights.
4. The method of claim 1, wherein processing the first intermediate representation comprises: Based on the first feature data, determine the target region for processing the first intermediate representation; as well as Based on the target region, the first intermediate representation is processed.
5. The method of claim 4, wherein determining the target region comprises: The feature data is divided into multiple sets of regional feature data, and the multiple sets of regional feature data correspond to multiple regions of the media content; Determine the measure of change of the multiple sets of regional feature data in a preset dimension; as well as The target region is determined based on the corresponding change measures of the multiple sets of regional feature data.
6. The method of claim 5, wherein the change measure comprises at least one of the following: A first measure of change in the spatial dimension, the first measure of change being determined based on feature values at multiple spatial locations within one of the plurality of regions. A second measure of change in the time dimension, which is determined based on feature values of the region across multiple time frames. A third measure of change in the channel dimension, which is determined based on the feature values of the region in multiple channels.
7. The method of claim 4, further comprising at least one of the following: The location data corresponding to the target area is sized to match the location data with the first intermediate representation in terms of spatial dimensions. Channel mapping is performed on the location data corresponding to the target area so that the location data matches the first intermediate representation in terms of the number of channels.
8. The method of claim 1, wherein processing the first intermediate representation based on the first direction data and the first feature data comprises: Based on the first direction data and the first feature data, determine the state deviation information; as well as Based on the state deviation information, the first intermediate representation is processed to obtain the second intermediate representation.
9. The method of claim 8, wherein processing the first intermediate representation based on the state deviation information comprises: Based on the first intermediate representation and the first direction data, a reference state is determined, wherein the reference state indicates the state of the first intermediate representation under the reference update mode; as well as Based on the baseline state and the state deviation information, the first intermediate representation is processed to obtain the second intermediate representation.
10. The method of claim 9, wherein processing the first intermediate representation based on the reference state and the state deviation information to obtain the second intermediate representation comprises: Based on the constraint strength controlling the regression of the first intermediate representation to the baseline state, the state deviation information, and the baseline state, state update data is determined, and the state update data indicates the update center position from the first intermediate representation to the second intermediate representation; Based on the constraint strength and random offset strength, random offset data is determined, the random offset data indicating the random offset range from the first intermediate representation to the second intermediate representation; and Based on the state update data and the random offset data, perform a state update on the first intermediate representation to obtain the second intermediate representation.
11. An apparatus for generating media content, comprising: The acquisition module is configured to acquire first direction data and first feature data, wherein the first direction data and the first feature data are obtained by the machine learning model processing the first intermediate representation in the same way, and the first intermediate representation is the intermediate state of the media content to be generated. The processing module is configured to process the first intermediate representation based on the first direction data and the first feature data to obtain a second intermediate representation; as well as The generation module is configured to generate the media content based on the second intermediate representation.
12. The apparatus of claim 11, wherein the processing module is further configured to: Based on the first direction data and the reference direction data, a target direction for processing the first intermediate representation is determined, wherein the reference direction data is determined based on at least one historical direction data of the machine learning model; and Based on the target direction, the first intermediate representation is processed.
13. The apparatus of claim 12, wherein the processing module is further configured to: Determine the weights of at least one historical direction data point, wherein the weight of the historical direction data point is inversely correlated with the time step of the historical direction data point relative to the first direction data point; and The reference direction data is determined based on the at least one historical direction data and the determined weights.
14. The apparatus of claim 13, wherein the processing module is further configured to: Based on the first feature data, determine the target region for processing the first intermediate representation; and Based on the target region, the first intermediate representation is processed.
15. An electronic device comprising: At least one processor; as well as At least one memory, coupled to the at least one processor and storing instructions for execution by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1 to 10.
16. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 10.
17. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 10.
