Methods, apparatus, devices, and computer storage media for generating audio and video.
By determining the channel-by-channel modulation amount in the audio and video generation model and injecting it into the audio generation branch, the problem of uncontrollable audio and video sound is solved, achieving controllable generation of timbre and enhancing the realism of audio and video, thus expanding the application scenarios.
Patent Information
- Application Number
- CN202611630752.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing audio-video joint generation models cannot achieve user-specified timbre control, resulting in uncontrollable audio and video sound.
The modulation amount is determined by obtaining reference audio and injected into the audio generation branch for adjustment. The denoised audio latent variables are generated by combining text description and initial noise, and denoised in parallel with the video generation branch. Finally, the target audio and video are decoded to ensure consistent timbre.
It achieves controllability of timbre in audio and video generation, reduces the risk of timbre leaking speech content, improves the realism of generated audio and video and the degree of adherence to text description, and broadens the scope of application of digital human video generation and virtual anchors.
Smart Images

Figure CN123123678A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video generation technology, and in particular to a method, apparatus, device, and computer storage medium for generating audio and video. Background Technology
[0002] With the rapid development of video technology, scenarios such as digital human video generation, virtual anchors, and film and short video dubbing and translation require that the generated characters not only have controllable appearances, but also controllable voice timbre. Since voice can reflect the identity information of a character to a certain extent, whether the reference timbre specified by the user can be faithfully reproduced directly determines whether the solution can be implemented.
[0003] In recent years, joint audio-video generation models have emerged that can simultaneously produce video of people and voice-over. These models typically employ a two-branch diffusion architecture, where video and audio are denoised in parallel by separate diffusion transformer branches. The two branches exchange information through cross-modal connections, and textual conditions are shared between them. The generated result is then decoded to output a video of a person with voice-over.
[0004] However, when generating audio and video, the above models often output arbitrary timbres from priors rather than user-specified timbres, making the sound in the generated audio and video uncontrollable. Summary of the Invention
[0005] This application provides a method, apparatus, device, and computer storage medium for generating audio and video, enabling the user to specify and control the sound timbre in the generated audio and video.
[0006] This application provides a method for generating audio and video, including: Obtain reference images, reference audio, and text descriptions; Based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio is determined, and the channel-wise modulation amount is used to adjust the channel-level features that characterize the timbre; The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation branch to obtain the denoised audio latent variables; The reference image, the text description, and the initialized video noise are injected into the video generation branch to obtain the denoised video latent variables. The audio latent variables and the video latent variables are decoded to obtain the target audio and the target video, wherein the timbre of the target audio is consistent with the timbre of the reference audio.
[0007] This application provides an audio / video generation apparatus, comprising: The first acquisition module is used to acquire reference images, reference audio, and text descriptions; The first determining module is used to determine, based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio, wherein the channel-wise modulation amount is used to adjust the channel-level features characterizing the timbre; The first injection module is used to inject the text description, the channel-by-channel modulation amount, and the initialized audio noise into the audio generation branch to obtain the denoised audio latent variables. The first injection module is further configured to inject the reference image, the text description, and the initialized video noise into the video generation branch to obtain the denoised video latent variables; The first processing module is used to decode the audio latent variable and the video latent variable to obtain the target audio and the target video, wherein the timbre of the target audio is consistent with the timbre of the reference audio.
[0008] This application provides an audio generation method, including: Obtain reference audio and text description; Based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio is determined, and the channel-wise modulation amount is used to adjust the channel-level features that characterize the timbre; The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation network to obtain the denoised audio latent variables; The audio latent variables are decoded to obtain the target audio, the timbre of which is consistent with that of the reference audio.
[0009] This application provides an audio generation apparatus, including: The second acquisition module is used to acquire reference audio and text description; The second determining module is used to determine, based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio, wherein the channel-wise modulation amount is used to adjust the channel-level features characterizing the timbre; The second injection module is used to inject the text description, the channel-by-channel modulation amount, and the initialized audio noise into the audio generation network to obtain the denoised audio latent variables. The second processing module is used to decode the audio latent variables to obtain the target audio, the timbre of which is consistent with the timbre of the reference audio.
[0010] This application provides an electronic device, including: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method.
[0011] This application provides a computer storage medium for storing a computer program that enables a computer to implement the above-described method when executed.
[0012] This application provides a computer program product, including: a computer program or instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the method described above.
[0013] The audio and video generation method, apparatus, device, and computer storage medium provided in this embodiment acquire a reference image, reference audio, and text description. Based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio is determined. Then, the text description, channel-wise modulation amount, and initialized audio noise are injected into the audio generation branch to obtain denoised audio latent variables. The reference image, text description, and initialized video noise are injected into the video generation branch to obtain denoised video latent variables. Finally, the audio latent variables and video latent variables are decoded to obtain the target audio and target video. In the above implementation process, on the one hand, timbre participates in the denoising process of the audio generation branch in the form of channel-wise modulation, so that the timbre of the generated target audio is no longer randomly determined by the generation prior, but determined by the reference audio specified by the user. This overcomes the problem of uncontrollable sound in the related dual-branch diffusion architecture and realizes controllable timbre generation. On the other hand, timbre is not concatenated or superimposed into the attention sequence in the form of tokens, but is applied to the mean and standard deviation of the features obtained by normalization through channel-level scaling and translation. That is, identity information can enter the generation process through global channel-wise statistics, without carrying the specific wording and rhythm carried in the reference audio. Therefore, identity and content are structurally constructed in the model structure. Decoupling eliminates the need for the model to learn decoupling during training, reducing the risk of timbre leakage of speech content and improving the degree of conformity of the generated audio to the text description. In addition, during the generation of audio and video, the audio generation branch and the video generation branch can perform parallel denoising and unified decoding according to their respective noise levels. The timbre modulation is applied only to the audio generation branch without disturbing the channel statistics of the video generation branch. Therefore, while ensuring the coordination between lip movements and speech, timbre control is achieved, improving the overall realism of the generated audio and video. This enables the joint generation of audio and video to faithfully reproduce the user-specified timbre with minimal structural cost, broadening the applicability of scenarios such as digital human video generation, virtual anchors, and dubbing and translation. Attached Figure Description
[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram illustrating a method for generating audio and video according to an exemplary embodiment of this application; Figure 2 A flowchart illustrating an audio / video generation method provided as an exemplary embodiment of this application; Figure 3 A schematic diagram illustrating the principle of an audio-video joint generation model provided as an exemplary embodiment of this application; Figure 4 A schematic diagram of a process for determining a channel-wise modulation amount corresponding to the timbre in a reference audio based on a reference audio, provided as an exemplary embodiment of this application; Figure 5 A flowchart illustrating the process of obtaining denoised audio latent variables provided for an exemplary embodiment of this application. Figure 1 ; Figure 6 A flowchart illustrating the process of obtaining denoised audio latent variables provided for an exemplary embodiment of this application. Figure 2 ; Figure 7 A flowchart illustrating another method for generating audio and video provided as an exemplary embodiment of this application; Figure 8 A schematic diagram of a process for retraining an audio generation branch to obtain a trained audio generation branch, provided as an exemplary embodiment of this application; Figure 9 A schematic diagram illustrating the principle of an audio / video generation method provided in an exemplary application embodiment of this application; Figure 10 A schematic diagram of the principle of a dual-branch diffusion network block provided in an exemplary application embodiment of this application; Figure 11 A schematic diagram illustrating the construction of an audio loss function provided in an exemplary application embodiment of this application; Figure 12 A schematic flowchart illustrating an audio generation method provided for an exemplary embodiment of this application; Figure 13 A schematic diagram of the structure of an audio / video generation apparatus provided for an exemplary embodiment of this application; Figure 14 A schematic diagram of the structure of an electronic device corresponding to an audio / video generation apparatus is provided for an exemplary embodiment of this application; Figure 15 A schematic diagram of an audio generation apparatus provided for an exemplary embodiment of this application; Figure 16 This application provides a schematic diagram of the structure of an electronic device corresponding to an audio generation apparatus, which is an exemplary embodiment of the present application. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0016] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, user profiles, historical behavior sequences, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, training data, sample data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.
[0017] Furthermore, in the cases where the embodiments of this application involve generating or editing the real-person voice and image corresponding to the reference audio and reference image, the acquisition and use of the aforementioned related materials have obtained separate authorization or consent from the subject to whom the voice and image belong, and comply with the requirements of relevant laws and regulations regarding the generation of synthetic content identifiers, such as embedding visual prompts or implicit digital identifiers in the obtained target audio and target video; the technical solutions of the embodiments of this application are not used in any scenario of impersonating others or forging others' statements.
[0018] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.
[0019] Terminology definition: Joint audio and video generation: refers to a generation method in which video and audio are produced simultaneously by the same generation model in the same generation process. For example, video and audio can be generated by two parallel branches that can exchange information.
[0020] Diffusion model / flow matching model: This refers to a generative model that reconstructs the target data distribution from the noise distribution through stepwise denoising or stepwise flow velocity integration. Correspondingly, the diffusion time step is used to characterize the current denoising stage.
[0021] Diffusion Transformer (DiT): This refers to a diffusion model that uses a transformer structure as the denoising backbone. It can inject conditional information through adaptive layer normalization.
[0022] A dual-branch diffusion network block refers to a cascaded denoising computation unit in the audio generation branch. A dual-branch diffusion network block includes at least a self-attention layer, a feedforward layer, and a normalization layer placed before both of these. For example, the feedforward layer may include a pointwise feedforward network, which is used to perform a channel-wise nonlinear transformation on the features.
[0023] Latent: refers to the low-dimensional continuous representation of an image, video frame or audio obtained by the encoder of a variational autoencoder (VAE); correspondingly, the audio latent variable and the video latent variable are the latent variables obtained by denoising the audio branch and the video branch, respectively, and the audio noise and video noise can be noise representations in the latent variable space.
[0024] Normalization layer: refers to a network layer that performs normalization operations on features along the channel dimension. For example, it can be a layer normalization layer. The normalization layer can apply an affine transformation to its normalized output through scaling and translation parameters.
[0025] Adaptive Layer Normalization (AdaLN) refers to the modulation mechanism that generates the scaling parameters, translation parameters, and residual gating parameters required for the normalization layer from external conditions.
[0026] Channel-by-channel modulation: refers to a set of modulation parameters given at the channel level, which may include channel-level scaling parameters, translation parameters and residual gating parameters; this channel-by-channel modulation is used to adjust the channel-level characteristics that characterize the timbre.
[0027] Learnable basis vectors are parameters that are inherent to each block of a two-branch diffuse network and are set independently on a block-by-block basis. They are used to reflect the differences between network blocks and between sub-layers within a network block.
[0028] Time step projection: refers to the modulation amount obtained after projecting the time step embedding determined by the diffusion time step.
[0029] Timbre vector: refers to a global, fixed-length vector representation of a reference audio extracted by a speaker representation network or other networks that can represent the timbre of a speaker. It is used to represent "who is speaking" and has no direct correspondence with the content being spoken.
[0030] Timbre extraction: refers to the process of obtaining the aforementioned timbre vector from the reference audio. For example, it can be a speaker representation network with the reference audio input parameters frozen and its output vector extracted.
[0031] Timbre offset: refers to the offset obtained by projecting the timbre vector into the same dimension as the time step projection. It can be fused with the time step projection and the learnable basis vector.
[0032] Projection network: refers to a network used to map fused offsets into multiple sets of channel-wise modulation quantities, such as a single-layer linear transform network or a multilayer perceptron.
[0033] Residual gating: refers to an operation method in which the output of a sub-layer is scaled by a gating parameter and then residually connected to the input of the sub-layer. The gating parameter is used to control the contribution of the sub-layer output to the degree of feature update.
[0034] Three-branch guidance refers to a guidance method that performs predictions based on three combinations of conditions: empty text with zero timbre, text with zero timbre, and text with timbre, and adjusts the degree of text compliance and timbre fidelity independently accordingly.
[0035] Zero timbre vector: refers to the vector obtained by setting the timbre offset in the channel-by-channel modulation to zero. It is used to characterize the conditional branch that does not provide timbre constraints.
[0036] Empty text description: refers to the conditional input obtained by setting the text description to empty, which is used to represent conditional branches that do not provide content constraints.
[0037] Test-time adaptation refers to a method of temporarily updating some parameters of the model during the inference phase for the current input sample in order to improve the generation effect on that sample.
[0038] Audio segment pair: refers to a pair of segments obtained by segmenting the audio of the same speaker, which includes a training audio segment used as conditional input and a ground truth audio segment used as the reconstruction target.
[0039] One-step denoising estimation refers to a closed-form estimation method that uses the current noisy latent variables and the noise or velocity field predicted by the model to estimate the latent variables of clean samples.
[0040] Digital human: refers to a virtual broadcasting character that is generated in a generative manner and possesses the image and voice of a target person.
[0041] To facilitate understanding of the audio and video generation method, apparatus, device, and computer storage medium provided in the embodiments of this application, the relevant technologies are briefly described below: With the rapid development of video technology, scenarios such as digital human video generation, virtual anchors, and dubbing and translation of films and short videos are constantly emerging. These scenarios require that the generated characters not only have controllable appearances but also controllable voice timbre. Currently, relevant implementation methods can include: Implementation method 1, based on live-action shooting and post-production editing.
[0042] The target person appears on camera to record and explain each segment of the video, which is then manually edited and subtitled to produce the final product.
[0043] The audio and video images and sounds obtained by this method are not distorted, but the production capacity is limited by the shooting time available for the target person. Moreover, when facing the needs of multiple categories and multiple scripts, repeated shooting is required, which is costly, slow to update, and difficult to scale up.
[0044] Implementation method 2, based on audio-driven lip-sync generation.
[0045] First, a speech synthesis model generates speech based on the text description, and then the speech drives the generation of a speaking face video based on the image of the target person.
[0046] While this implementation method has a certain ability to reuse sound, its lip movements are matched post-processed by an independent driver module, and the video and audio are not jointly modeled. Therefore, in scenarios of rapid pronunciation and continuous speech, the lip movements and speech may not match in time.
[0047] Implementation method 3: Dual-branch diffusion implementation method with simultaneous audio and video output.
[0048] This implementation sets up two diffusion transformer branches, one for video and one for audio, for parallel denoising. The two branches exchange information through a cross-modal connection. The text conditions are shared by the two branches. The generated result is decoded and output as a video of a person with voice.
[0049] This implementation improves the temporal coordination between audio and video. However, the output timbre of its audio branch is randomly determined by the generation prior. The model will emit any timbre from the prior instead of the timbre specified by the user, which makes the sound in the generated audio and video uncontrollable.
[0050] Implementation method 4: Injecting reference timbres onto the lexical axis.
[0051] To address the aforementioned issue of uncontrollable sound, this implementation attempts to inject reference timbres into the audio branches. Specifically, the following methods can be used: 1) Concatenation-based injection, which encodes the reference speech into several additional lexical units and concatenates them into the sequence being attended to; 2) Additive injection, which involves directly superimposing the pooled speaker global vector onto the word stream; 3) Parallel auxiliary branch injection, that is, starting another network branch to process the reference audio, and then merging it back into the main branch in the form of gated residuals.
[0052] It should be noted that the above implementation methods still struggle to reliably achieve controllable timbre. Specifically, timbre is essentially a global attribute, independent of the spoken content. However, the aforementioned implementation methods all inject timbre onto the lexical axis, forcing global attributes to participate in calculations on a lexical basis. For concatenative injection and parallel auxiliary branch injection, the specific wording and prosody in the reference speech are noticed by the model along with the speaker's identity, posing a risk of content leakage. Furthermore, the decoupling of speaker identity and speech content needs to be learned by the model during training, resulting in unstable decoupling effects. Additionally, the increased length of the noticed sequence leads to higher memory and computational costs, or significantly increases parameter and structural complexity. For additive injection, the superimposed bias is absorbed or weakened by subsequent normalization operations, making it difficult to reliably change the first and second-order statistics at the channel level, thus resulting in weaker timbre control.
[0053] As can be seen from the above, the relevant technologies either lack the ability to specify the output timbre, or although they attempt to specify the timbre, problems such as the entanglement of identity and content due to injection on the lexical axis, easy content leakage, expansion of sequence length or parameter scale, and insufficient control make the sound in the generated audio and video still uncontrollable.
[0054] To address the aforementioned issues, this application provides an audio / video generation method, apparatus, device, and computer storage medium. The audio / video generation method operates as follows: instead of providing the timbre of the reference audio as a lexical condition to the generation model, it determines a channel-wise modulation amount based on the reference audio to adjust the timbre features during the denoising process. This channel-wise modulation amount is then injected into the channel-level statistics used for normalization operations in the audio generation branch, allowing the timbre to be incorporated into the audio / video generation process through global, channel-wise scaling and translation. This ensures that the timbre of the target audio is truly constrained by the reference audio, and that identity information is structurally limited to be expressible through channel statistics. This decouples identity from content during audio / video generation without increasing the length of the attention sequence or introducing additional parameters.
[0055] For details, please refer to Figure 1As shown, the entity executing this audio / video generation method can be an audio / video generation device 200, which can be implemented as a local server, a cloud server, etc. When the audio / video generation device 200 is implemented as a cloud server, the audio / video generation method can be executed in the cloud. Several computing processes (cloud servers) can be deployed in the cloud, each with processing resources such as computing and storage. In the cloud, multiple computing processes can be organized to provide a certain service; of course, a single computing process can also provide one or more services. The cloud can provide this service by providing a service interface, which users call to use the corresponding service. Service interfaces include Software Development Kits (SDKs), Application Programming Interfaces (APIs), etc.
[0056] The audio / video generation device 200 can communicate with the requesting terminal 100, which is used by users to trigger audio / video generation operations. The requesting terminal 100 can be any computing device with certain information interaction capabilities; specifically, it can be a mobile phone, a personal computer (PC), a tablet computer, a setting application, etc. Furthermore, the basic structure of the requesting terminal 100 can include at least one processor. The number of processors depends on the configuration and type of the requesting terminal 100. The requesting terminal 100 may also include memory, which can be volatile, such as Random Access Memory (RAM), or non-volatile, such as Read-Only Memory (ROM), flash memory, or both. The memory typically stores the operating system (OS), one or more applications, and may also store program data. In addition to the processing unit and memory, the requesting terminal 100 also includes some basic configurations, such as a network interface card (NIC) chip, an input / output (I / O) bus, a display component, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, mouse, stylus, printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.
[0057] The audio / video generation device 200 refers to a device capable of generating audio and video in a network virtual environment. It typically refers to a device that utilizes a network for information planning and audio / video generation. Physically, the audio / video generation device 200 can be any device capable of providing computing services and performing corresponding audio / video generation operations, such as a processor or server. The audio / video generation device 200 mainly consists of a processor, hard disk, memory, and system bus, and its architecture is similar to that of a general-purpose computer.
[0058] In the above embodiment, the audio / video generation device 200 and the requesting terminal 100 are connected via a network, which can be a wireless or wired network connection. If the audio / video generation device 200 and the requesting terminal 100 are connected via a communication connection, the network standard of the mobile network can be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA)), 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), Global Microwave Access Interoperability (WiMax), 5G, 6G, etc.
[0059] In this embodiment of the application, the requesting end 100 runs an audio and video generation application. The user uploads reference images and reference audio through the audio and video generation application, and inputs a text description to describe the video content and audio content to be generated. The requesting end 100 sends the above-mentioned reference images, reference audio and text description to the audio and video generation device 200 through the network.
[0060] The audio / video generation device 200 can be a standalone physical server, a server cluster consisting of multiple servers, or a distributed system. It can also be a platform server providing cloud computing services. The requesting end 100 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, smart voice interaction devices, and smart vehicle terminals. The audio / video generation device 200 deploys an audio / video joint generation model. The model parameters in this model can be stored in a database 300 and loaded by the audio / video generation device 200, or the audio / video generation device 200 can save reference data to the database 300 after training. The audio / video generation device 200 uses the audio / video joint generation model to generate target audio and target video, and returns the target audio and target video as digital human audio / video to the requesting end 100 for display via the network. It should be noted that the audio / video generation method provided in this application can also be executed entirely or partially by the requesting end 100 locally; this embodiment does not limit this.
[0061] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0062] refer to Figure 2 As shown, this embodiment provides a method for generating audio and video. The execution subject of this method is an audio and video generation device. The audio and video generation device can be implemented as software or a combination of software and hardware. When the audio and video generation device is implemented as hardware, it can specifically be various electronic devices capable of performing audio and video generation operations, including but not limited to software vendors' clients, servers, etc. When the audio and video generation device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the audio and video generation method may include: Step S201: Obtain the reference image, reference audio, and text description.
[0063] Step S202: Based on the reference audio, determine the channel-by-channel modulation amount corresponding to the timbre in the reference audio. The channel-by-channel modulation amount is used to adjust the channel-level features that characterize the timbre.
[0064] Step S203: Inject the text description, channel-by-channel modulation amount, and initialized audio noise into the audio generation branch to obtain the denoised audio latent variables.
[0065] Step S204: Inject the reference image, text description, and initialized video noise into the video generation branch to obtain the denoised video latent variables.
[0066] Step S205: Decode the audio latent variables and video latent variables to obtain the target audio and target video. The timbre of the target audio is consistent with the timbre of the reference audio.
[0067] The specific implementation principles and effects of each of the above steps are explained in detail below: Step S201: Obtain the reference image, reference audio, and text description.
[0068] When there is a need to generate audio and video, the audio and video generation device can acquire reference images, reference audio, and text descriptions. The reference images are used to determine the image in the target video to be generated. For example, the reference image can be an image containing a person's face, or it can be an image containing a person's image and a scene. This embodiment does not limit the content or number of reference images. The reference audio is used to determine the timbre of the target audio to be generated; it is a piece of speech that can reflect the timbre characteristics of the target speaker. For example, the reference audio can be a few seconds of reading or dialogue. The text description is used to determine the visual content of the target video and the audio content of the target audio. For example, the text description can include descriptions of character actions, scenes, expressions, and descriptions of lines to be spoken. The aforementioned reference images, reference audio, and text descriptions can be obtained by the user through a request or input, or they can be obtained by the audio and video generation device from a database. This embodiment does not limit their source.
[0069] Step S202: Based on the reference audio, determine the channel-by-channel modulation amount corresponding to the timbre in the reference audio. The channel-by-channel modulation amount is used to adjust the channel-level features that characterize the timbre.
[0070] First, by extracting the mean and variance of each channel in the latent space of audio from different speakers and with different content, and then visualizing the results through dimensionality reduction, the results can be clustered into several clusters according to the speaker, with little relation to the content. This indicates that the channel-level statistics of audio latent variables can effectively carry timbre information. Based on this, this embodiment can specify the timbre by adjusting the channel statistics without changing the word structure or increasing the length of the attention sequence, so that the timbre of the target audio can stably follow the reference audio.
[0071] After obtaining the reference audio, it can be analyzed to determine the channel-wise modulation amount corresponding to the timbre in the reference audio. In one possible implementation, the channel-wise modulation amount can be obtained through a projection mapping operation using a projection network. In this case, determining the channel-wise modulation amount corresponding to the timbre in the reference audio based on the reference audio may include: retrieving the projection network used for analyzing and processing the reference audio; inputting the reference audio into the projection network for projection operation to obtain the channel-wise modulation amount output by the projection network corresponding to the timbre in the reference audio.
[0072] Specifically, after inputting the reference audio into the projection network, the projection network first performs timbre extraction on the reference audio to obtain a timbre vector representing the timbre. Then, it transforms the timbre vector into an offset form that matches the modulation interface within the denoising network, thereby obtaining the channel-wise modulation amount. The determined channel-wise modulation amount is a modulation amount represented at the channel level. It does not act on the word axis, but rather on the channel statistics of the normalization layer in the audio-video joint generation model. In other words, the channel-wise modulation amount is used to adjust the distribution of features along the channel dimension during the denoising process.
[0073] Step S203: Inject the text description, channel-by-channel modulation amount, and initialized audio noise into the audio generation branch to obtain the denoised audio latent variables.
[0074] refer to Figure 3 As shown, a pre-configured audio-video joint generation model is provided for implementing audio and video generation operations. This model includes a video generation branch, an audio generation branch, a video decoder, and an audio decoder, etc. Both the video generation branch and the audio generation branch can be implemented using a diffusion model or a stream matching model; for example, a Diffusion Transformer (DiT) can be used.
[0075] Both the video generation branch and the audio generation branch can include multiple cascaded network blocks, each containing structures such as a normalization layer, a self-attention layer, and a feedforward layer. The audio generation branch and the video generation branch exchange information bidirectionally through cross-modal connections. For example, in the attention operation of a network block on one side, the cross-modal connection can use the output of the corresponding network block on the other side as the key value information being paid attention to, thereby achieving mutual conditional constraints between audio and video terms. The cross-modal connection can also use methods such as rotational position encoding to align the time axes on both sides to ensure the temporal correspondence between lip movements and pronunciation. This embodiment does not limit the specific implementation method of the cross-modal connection.
[0076] The denoising result obtained from the video generation branch (i.e., the denoised video latent variables) is input into the video decoder to obtain the target video; the denoising result obtained from the audio generation branch (i.e., the denoised audio latent variables) is input into the audio decoder to obtain the target audio. For example, the video decoder can use the decoding end of a pre-trained variational autoencoder (VAE), and the audio decoder can use the decoding end of an audio codec or a vocoder.
[0077] For example, the audio-video joint generation model may further include a channel-wise modulation quantity providing module (not shown in the figure). This module takes reference audio as input and outputs a channel-wise modulation quantity corresponding to the timbre. It can also inject text descriptions, channel-wise modulation quantities, and initialized audio noise into the audio generation branch. Specifically, the channel-wise modulation quantities can be injected into the normalization layer of each network block in the audio generation branch to obtain the denoised audio latent variables. In the illustrated embodiment, the channel-wise modulation quantities are only injected into the audio generation branch and not into the video generation branch. That is, specifying the timbre does not change the computation process and generation result of the video generation branch, thus achieving controllable timbre without affecting the picture quality and image consistency of the target video.
[0078] Step S204: Inject the reference image, text description, and initialized video noise into the video generation branch to obtain the denoised video latent variables.
[0079] After obtaining the reference image and text description, the reference image, text description, and initialized video noise can be injected into the video generation branch to obtain the denoised video latent variables. The initialized audio noise and initialized video noise can be noise data sampled from a preset noise distribution (e.g., a standard normal distribution), with dimensions identical to the dimensions of the audio and video latent variables, respectively.
[0080] Furthermore, the audio generation branch and the video generation branch can execute the denoising process in parallel, and exchange information bidirectionally through cross-modal connections during the denoising process to ensure that the generated audio and video are temporally aligned and content-matched. It should be noted that steps S203 and S204 do not have a strict order; they can be executed simultaneously as two branches within the same denoising iteration.
[0081] Step S205: Decode the audio latent variables and video latent variables to obtain the target audio and target video. The timbre of the target audio is consistent with the timbre of the reference audio.
[0082] Specifically, the denoised audio latent variables can be input into an audio decoder for waveform reconstruction to obtain the target audio in the time domain; the denoised video latent variables can be input into a video decoder for pixel reconstruction to obtain the target video. Since the denoising process in the audio generation branch is modulated by the channel-by-channel modulation, the generated target audio tends to have the timbre of the reference audio, while the picture content of the target video is determined by the reference image and text description, thus obtaining a digital human audio-visual system where both image and sound are controllable.
[0083] In addition, the consistency between the timbre in the target audio and the timbre in the reference audio means that the similarity between the timbre features in the target audio and the timbre features in the reference audio is greater than or equal to a preset similarity threshold. The similarity threshold can be 90% or 95%, etc. That is, the consistency between the timbre in the target audio and the timbre in the reference audio can include: the case where the timbre in the target audio and the timbre in the reference audio are the same, and the case where the timbre in the target audio and the timbre in the reference audio have a high degree of similarity.
[0084] It is worth noting that during the generation of target audio and target video, the speech content in the reference audio does not participate in the text conditional injection of the audio generation branch. The speech content of the target audio is determined by the text description, and the timbre of the target audio is determined by the per-channel modulation. In addition, during the generation of target audio and target video, the per-channel modulation is not used as an additional lexical unit in the attention operation, nor is it simply superimposed on the lexical stream. Instead, it directly affects the channel statistics output by the normalization layer. Therefore, structurally, the global attribute of timbre is isolated from the lexical axis containing the speech content and the visual content.
[0085] The audio and video generation method provided in this embodiment incorporates timbre into the denoising process of the audio generation branch in the form of channel-wise modulation. This ensures that the timbre of the generated target audio is no longer randomly determined by the prior generation, but rather by the reference audio specified by the user. This overcomes the problem of uncontrollable sound in the related dual-branch diffusion architecture and achieves controllable timbre generation. Furthermore, the timbre is not concatenated or superimposed onto the attention sequence in the form of tokens, but rather applied to the mean and standard deviation of the features obtained from normalization through channel-level scaling and translation. In other words, identity information can enter the generation process through global channel-wise statistics, without carrying the specific wording and rhythm carried in the reference audio. Therefore, identity and content are constructed in the model structure. The decoupling is achieved automatically, without relying on the model to learn decoupling during training, reducing the risk of timbre leakage of speech content and improving the degree of conformity of the generated audio to the text description. In addition, during the generation of audio and video, the audio generation branch and the video generation branch can be denoised in parallel and decoded uniformly according to their respective noise levels. The timbre modulation is only applied to the audio generation branch without disturbing the channel statistics of the video generation branch. Therefore, while ensuring the coordination between lip movements and speech, the timbre is controllable, improving the overall realism of the generated audio and video. This achieves the ability to faithfully reproduce the user-specified timbre in joint audio and video generation with minimal structural cost, broadening the applicability of scenarios such as digital human video generation, virtual anchors, and dubbing and translation.
[0086] Based on the above embodiments, refer to Figure 4As shown, when the audio generation branch includes multiple cascaded bi-branch diffusion network blocks, a projection operation can be performed by combining the learnable basis vectors of the bi-branch diffusion network blocks, the reference audio, and the time step to obtain the channel-wise modulation amount. In this case, determining the channel-wise modulation amount corresponding to the timbre in the reference audio, based on the reference audio, can include: Step S401: Extract the timbre from the reference audio to obtain the timbre vector.
[0087] In one alternative implementation, extracting the timbre of the reference audio to obtain the timbre vector may include: inputting the reference audio into a pre-trained speaker representation network (or speaker verification network / speaker recognition network / speaker embedding network) to obtain the timbre vector output by the speaker representation network, wherein the speaker representation network is a knowledge network trained to distinguish speakers, and the parameters of the speaker representation network are frozen.
[0088] Specifically, the reference audio input is used to represent the speaker's timbre in a speaker representation network, and the timbre vector output by the speaker representation network is obtained. For example, the speaker representation network can be a parameter-frozen speaker representation network, such as: an x-vector network based on short-time extension, a d-vector network based on clustering, a CAM++ network, an ECAPA-TDNN network, an ERes2Net network, a large model speech encoder, etc. Alternatively, the intermediate layer output of a self-supervised speech model (such as WavLM, HuberT, etc.) can be used as the timbre vector. This embodiment does not limit the type of speaker representation network.
[0089] The timbre vector obtained above can be a global, fixed-length vector. For example, the timbre vector can be a one-dimensional vector (e.g., a single vector with dimension 512). For example, a reference audio corresponds to a global, fixed-length timbre vector, which means that the timbre information does not carry word-level content information at the source.
[0090] In another alternative implementation, extracting timbre from the reference audio to obtain a timbre vector may include: encoding the reference audio using a variational autoencoder to obtain latent variables of the reference audio; pooling the latent variables of the reference audio along the time dimension to obtain first-order and second-order statistics per channel; and constructing a timbre vector from the first-order and second-order statistics per channel.
[0091] In another alternative implementation, extracting timbre from the reference audio to obtain a timbre vector may include: encoding the reference audio using a variational autoencoder and pooling the resulting latent variables of the reference audio along the time dimension to obtain first-order and second-order statistics for each channel; fusing the timbre vector output by the speaker representation network with the offset timbre vector formed by the first-order and second-order statistics to obtain the final timbre vector, wherein the fusion operation may include concatenation or weighted summation.
[0092] Step S402: Project the timbre vector into a timbre offset. The dimension of the timbre offset is the same as the dimension of the time step projection. The time step projection is determined based on the diffusion time step.
[0093] After obtaining the timbre vector, it can be projected into a timbre offset. It's important to note that the timbre offset obtained from the projection operation has the same dimension as the time-step projection; that is, the timbre vector and the time-step projection have the same dimensional shape, allowing them to be fused in the same addition operation. The time-step projection can be determined through the following steps: first, determine the diffusion time step, and obtain the time-step embedding from the diffusion time step; then, perform projection on the time-step embedding to obtain the time-step projection.
[0094] For example, assuming the hidden layer width is D, the modulation required for each dual-branch diffusion network block is a 6×D dimensional vector (corresponding to two sets of scaling, translation, and residual gating, each with a width of D). The timbre offset can be projected into the same shape as this 6×D dimension. It should be noted that the timbre vector projection in this step is a single-layer linear transformation, and its parameter count is extremely small relative to the total parameter count of the audio-video joint generation model, thus hardly increasing the model's parameter size.
[0095] Step S403: Fuse the timbre offset, time step projection, and learnable basis vectors of the bi-branch diffusion network block to obtain the fused offset.
[0096] In one feasible approach, fusing the timbre offset, time step projection, and learnable basis vectors may include: adding the timbre offset, time step projection, and learnable basis vectors element by element; or concatenating the three and then projecting them to obtain the fused offset; or applying the timbre offset as an offset independent of the time step projection in parallel to the normalization layer; or gating and weighting the timbre offset and time step projection, and then adding it to the learnable basis vectors to obtain the fused offset.
[0097] For example, let the first... The learnable basis vectors of each bi-branch diffuse network block are m^( Let MLP_t(e_t) be the projection of the time step e_t, and MLP_s(e_spk) be the timbre offset corresponding to the timbre vector e_spk. Then the blending offset can be expressed as: m^( )+MLP_t(e_t)+MLP_s(e_spk); where MLP_s is the speaker projection network and MLP_t is the time-step projection network that already exists in the audio-video joint generation model.
[0098] It should be noted that the time step projection and timbre offset can be shared among all or multiple two-branch spread network blocks. Different two-branch spread network blocks correspond to different learnable basis vectors; that is, the differences between two-branch spread network blocks can be represented by their respective learnable basis vectors m^( For example, learnable basis vectors can be understood as parameters used to represent the differences between various network layers (including deep and shallow layers), with different network layers corresponding to different learnable basis vectors.
[0099] In the above implementation process, by adding a timbre offset on the basis of the existing time step projection, the two types of global conditions, "who is speaking" and "to what point in the speech", are entered into the audio and video joint generation model in the same form (i.e., channel-by-channel modulation), thereby reusing the original mechanism of adaptive layer normalization without changing the structure of the main network.
[0100] Step S404: Project the fusion offset to obtain the channel-by-channel modulation amount.
[0101] The channel-wise modulation amount can be obtained by projecting the fusion offset through a projection network. This projection operation can include: calling the projection network to project the fusion offset to obtain the channel-wise modulation amount. Specifically, the projection network can output multiple sets of channel-wise modulation amounts corresponding to multiple normalization layers in the audio generation branch; that is, only one projection is needed to obtain the modulation amounts required for each normalization layer distributed within each dual-branch diffusion network block. For example, the output dimension of the projection network can be an integer multiple of the hidden layer width in the audio generation branch, and the projection network can be a single-layer linear transform network or a multilayer perceptron. The projection network can share parameters across blocks or be set independently per block; this embodiment does not limit this.
[0102] It is worth noting that, because the fusion offset includes global time-step projection, global timbre offset, and static learnable basis vectors, the obtained channel-by-channel modulation is consistent across all word positions and does not exhibit word-level differences with variations in specific speech content. Therefore, the timbre vector and text content are separated at the parameter shape level, meaning the timbre vector primarily reflects channel statistics rather than the specific content of certain words.
[0103] In this embodiment, the timbre is extracted from the reference audio to obtain a timbre vector. The timbre vector is then projected into a timbre offset. The timbre offset, the time step projection, and the learnable basis vector of the dual-branch diffusion network block are then fused to obtain a fused offset. The fused offset is then projected into a channel-wise modulation amount. This effectively achieves the accurate determination of the channel-wise modulation amount.
[0104] Based on the above embodiments, refer to Figure 5 As shown, this embodiment provides a method for obtaining denoised audio latent variables using a normalization layer. In this case, injecting text description, channel-by-channel modulation, and initialized audio noise into the audio generation branch, the obtained denoised audio latent variables may include: Step S501: Inject channel-by-channel modulation into at least one normalized layer in the dual-branch diffuse network block to obtain the normalized layer after injection parameters.
[0105] In this embodiment, the audio generation branch is used to generate audio based on text description and timbre vector injected through a normalization layer. The audio generation branch may include multiple cascaded dual-branch diffusion network blocks. Any one of the dual-branch diffusion network blocks adopts a pre-normalization layout. Specifically, any dual-branch diffusion network block includes at least: a self-attention layer, a feedforward layer (which may be a point-by-point feedforward network), a normalization layer before the self-attention layer, and a normalization layer before the feedforward layer. Scaling parameters and translation parameters are applied to the normalization layer before the self-attention layer and the normalization layer before the feedforward layer, respectively. Residual gating parameters are applied to the output of the corresponding dual-branch diffusion network block.
[0106] After obtaining the channel-by-channel modulation amount, the determined channel-by-channel modulation amount can be distributed to the normalization layer of each dual-branch diffusion network block in the audio generation branch to obtain the normalization layer after injection parameters, so that each normalization layer after injection parameters can perform calculations with the scaling parameters, translation parameters and residual gating parameters provided.
[0107] Specifically, the normalization layers mentioned above are modulated by their respective channel-wise modulation amounts. The normalization layer before the attention layer can receive the scaling parameter γ_msa, the translation parameter β_msa, and the residual gating parameter α_msa; the normalization layer before the feedforward layer can receive the scaling parameter γ_mlp, the translation parameter β_mlp, and the residual gating parameter α_mlp. That is, a dual-branch diffusion network block in the audio generation branch requires a total of six sets of channel-wise modulation amounts, which are obtained by projecting the fusion offset mentioned above.
[0108] For example, the channel-by-channel modulation can be injected into all the dual-branch diffusion network blocks in the audio generation branch, or it can be injected into only some of the network layers (e.g., only the deep network blocks). This embodiment does not limit the number and location of the injected network layers.
[0109] Step S502: Inject the text description, channel-by-channel modulation amount and initial audio noise into the audio generation branch, and use the normalization layer after injection parameters to modulate the timbre features in the denoising process to obtain the denoised audio latent variables.
[0110] In one feasible approach, the channel-wise modulation amount can include at least: channel-level scaling parameters, translation parameters, and residual gating parameters. In this case, modulating the timbre features in the denoising process using the normalization layer after injecting parameters to obtain the denoised audio latent variables can include: scaling and translating the standard deviation and mean of the features obtained after normalization along the channel dimension using the scaling and translation parameters in the normalization layer after injecting parameters to obtain the layer output features; and gating the update degree corresponding to the layer output features based on the residual gating parameters to obtain the denoised audio latent variables.
[0111] After injecting the text description, channel-by-channel modulation, and initial audio noise into the audio generation branch, the scaling and translation parameters in the normalization layer after injection can be used to scale and translate the standard deviation and mean of the features obtained after normalization along the channel dimension. Specifically, the scaling and translation operations can be implemented using the following formula: h'_i=(1+γ_i)⊙LayerNorm(h_i)+β_i; In the above formula, h_i represents the features before entering the i-th normalization layer, LayerNorm(·) is the normalization operation, γ_i and β_i are the scaling and translation parameters for each channel, ⊙ represents element-wise multiplication along the channel dimension, and h'_i is the layer output feature corresponding to the double-branch diffuse network block.
[0112] After obtaining the layer output features, a gating operation can be performed on the update degree corresponding to the layer output features based on the residual gating parameters to obtain intermediate features in the denoising process. These intermediate features are then gradually denoised through multiple cascaded bi-branch diffusion network blocks to obtain the denoised audio latent variables. Specifically, the gating operation can be implemented using the following formula: h_{i+1}=h_i+α_i⊙SubLayer_i(h'_i).
[0113] In the above formula, h_i represents the features before entering the i-th normalization layer, ⊙ represents element-wise multiplication along the channel dimension, α_i is the residual gating parameter, SubLayer_i(·) represents the self-attention operation or feedforward operation, h_{i+1} is the intermediate feature of this double-branch diffusion network block, and h'_i is the layer output feature corresponding to the double-branch diffusion network block. After obtaining the intermediate features of this double-branch diffusion network block, further analysis and processing of the intermediate features can be performed to obtain the denoised audio latent variables.
[0114] It should be noted that the normalization layer in this embodiment does not carry learnable affine parameters. That is, the channel-wise scaling factor (1+γ_i) and channel-wise translation factor β_i that actually reach the audio lexical units in the above formula are determined by the aforementioned fusion offset. Thus, changing the reference timbre directly changes these two first-order and second-order statistical factors without altering the semantics of the content lexical units. This achieves decoupling between timbre information and content in the model construction, rather than relying on the model to learn the distinction between the two during training. Specifically, the timbre features are directly applied to the statistics of the normalization operation, thereby improving the stability of timbre control.
[0115] In this embodiment, scaling and translation parameters are used to adjust the standard deviation and mean of the features after normalization, so that the standard deviation and mean of the adjusted timbre features are close to the standard deviation and mean of the timbre features in the reference audio. In addition, the residual gating parameter is used to determine the contribution of the normalization layer output to feature updates. Therefore, even if the timbre modulation injected in a certain denoising stage is strong, the audio and video joint generation model can still suppress its influence on the main path through gating, thereby ensuring the stability of the generation of latent variables of the denoised audio.
[0116] Based on the above embodiments, refer to Figure 6 As shown, when generating denoised audio latent variables, not only can the channel-wise modulation amount be directly injected into the normalization layer in the dual-branch diffusion network block to generate denoised audio latent variables, but a three-branch denoising guided method can also be used to generate denoised audio latent variables. In this case, the text description, channel-wise modulation amount, and initialized audio noise are injected into the audio generation branch, and the denoised audio latent variables obtained can include: Step S601: Determine the zero timbre vector corresponding to the channel-by-channel modulation amount and the empty text description corresponding to the text description. The zero timbre vector is obtained by setting the timbre offset of the channel-by-channel modulation amount to zero, and the empty text description is obtained by setting the text description to empty.
[0117] In the process of generating denoised audio latent variables using the audio generation branch, after injecting the text description, channel-wise modulation amount and initial audio noise into the audio generation branch, the channel-wise modulation amount can be set to zero with a preset probability. The preset probability can take effect during the model training phase. For example, the preset probability can be any value between 5% and 15% (e.g., 10%). This value can be set manually according to experience. This embodiment does not limit its specific value.
[0118] Specifically, in the inference process of generating denoised audio latent variables using the audio generation branch, setting the timbre vector to zero with a preset probability while retaining the text description yields a zero timbre vector. Similarly, setting the text description to empty with a preset probability while retaining the timbre vector yields an empty text description. This achieves the three branches of the inference stage: an unconditional branch consisting of a zero timbre vector and an empty text description; a text-conditional branch containing only the text description; and a biconditional branch containing both the text description and the timbre vector.
[0119] It should be noted that by randomly setting the channel-by-channel modulation to zero and the text description to empty with a preset probability, the model can be equipped with the ability to guide the timbre conditions and text conditions separately during the inference stage, thus providing a basis for independent adjustment during the inference stage.
[0120] Step S602: Using the audio generation branch, the text description, channel-by-channel modulation amount, initialized audio noise, zero timbre vector, and empty text description are used as conditions to generate denoised audio latent variables.
[0121] In one feasible approach, after obtaining the text description, channel-wise modulation amount, initialized audio noise, zero timbre vector, and empty text description, the text description, channel-wise modulation amount, initialized audio noise, zero timbre vector, and empty text description can be input into the audio generation branch to obtain the denoised audio latent variables output by the audio generation branch. In this way, the denoised audio latent variables can be generated stably.
[0122] In another feasible approach, using the audio generation branch to generate denoised audio latent variables by taking the text description, channel-wise modulation amount, initial audio noise, zero-timbre vector, and empty text description as conditions can include: generating a first latent variable by taking the empty text description, zero-timbre vector, and initial audio noise as conditions by taking the audio generation branch; generating a second latent variable by taking the text description, zero-timbre vector, and initial audio noise as conditions by taking the audio generation branch; generating a third latent variable by taking the text description, channel-wise modulation amount, and initial audio noise as conditions by taking the audio generation branch; and determining the denoised audio latent variables based on the first, second, and third latent variables.
[0123] To highlight the impact of timbre vectors and text descriptions on the generation process of latent variables in the denoised audio, we can first input empty text descriptions, zero timbre vectors, and initialized audio noise as conditions into the audio generation branch to generate the first latent variable; similarly, we can input text descriptions, zero timbre vectors, and initialized audio noise as conditions into the audio generation branch to generate the second latent variable; and we can input text descriptions, channel-wise modulation amounts, and initialized audio noise as conditions into the audio generation branch to generate the third latent variable.
[0124] After obtaining the first, second, and third latent variables, they can be analyzed and processed to determine the denoised audio latent variables. In some instances, determining the denoised audio latent variables based on the first, second, and third latent variables may include weighted summation of the first, second, and third latent variables to obtain the denoised audio latent variables.
[0125] In another possible approach, determining the denoised audio latent variables based on the first latent variable, the second latent variable, and the third latent variable may include: determining the first difference information between the second latent variable and the first latent variable, and the second difference information between the third latent variable and the second latent variable; and performing a weighted summation of the first latent variable, the first difference information, and the second difference information to obtain the denoised audio latent variables.
[0126] Specifically, since the third latent variable is generated by combining text description, channel-by-channel modulation, and initial audio noise as conditions, the first latent variable is generated by combining empty text description, zero timbre vector, and initial audio noise as conditions, and the second latent variable is generated by combining text description, zero timbre vector, and initial audio noise as conditions, to highlight the influence of text description on the generation of audio latent variables, the first and second latent variables can be compared to determine the first difference information between the second and first latent variables. Similarly, to highlight the influence of timbre vector on the generation of audio latent variables, the third latent variable can be compared with the second latent variable to determine the second difference information between the third and second latent variables.
[0127] After determining the first difference information and the second difference information, a weighted summation operation can be performed on the first latent variable, the first difference information, and the second difference information to obtain the denoised audio latent variables. For example, performing a weighted summation operation on the first latent variable, the first difference information, and the second difference information to obtain the denoised audio latent variables may include: determining the first guiding weight corresponding to the first difference information and the second guiding weight corresponding to the second difference information; and then performing a weighted summation operation on the first latent variable, the first difference information, and the second difference information based on the first guiding weight and the second guiding weight to obtain the denoised audio latent variables.
[0128] The first guiding weight is used to adjust the degree of conformity of the generated audio to the text description, and the second guiding weight is used to adjust the timbre fidelity of the generated audio relative to the reference audio. The first guiding weight and the second guiding weight can be set independently of each other.
[0129] For example, if we take Let v_text represent the predicted value corresponding to the first latent variable, v_cond represent the predicted value corresponding to the second latent variable, and v_text represent the predicted value corresponding to the third latent variable. Then, the modulated audio latent variable v can be expressed as: v= +s_text·(v_text- )+s_spk·(v_cond-v_text); Here, `s_text` is the first guiding weight, used to adjust the degree of fit between the generated audio and the text description; `s_spk` is the second guiding weight, used to adjust the timbre fidelity between the generated audio and the reference audio. It should be noted that `s_text` and `s_spk` are weight coefficients that can be set independently. Neither needs to satisfy normalization constraints, and either weight coefficient can be zero. For example, when `s_text` is zero, it is equivalent to abandoning the additional enhancement to the text fit.
[0130] In this embodiment, by configuring the unconditional branch (used to generate the first latent variable), the text-only branch (used to generate the second latent variable), and the complete conditional branch (used to generate the third latent variable) as a series of two levels of differences, the text fit and timbre fidelity are made to have two mutually orthogonal adjustable dimensions, which further improves the controllability of timbre in audio and video generation.
[0131] Based on any of the above embodiments, considering the uneven distribution of sample sizes corresponding to different speakers in the training data, and the insufficient learning level of some speakers' timbre during the training phase, the timbre of the output target audio after one generation may still deviate from the reference audio. To improve this situation, this embodiment also provides a model retraining process after obtaining the target audio and target video, referring to... Figure 7 As shown, after obtaining the target audio and target video, the method in this embodiment may further include: Step S701: Determine the timbre similarity between the target audio and the reference audio.
[0132] After obtaining the target audio and target video, the target audio and reference audio can be analyzed and processed to determine the timbre similarity between them. Specifically, the timbre embeddings of the target audio and the reference audio can be extracted using a speaker representation network, and the similarity between the two can be determined as the timbre similarity. For example, the similarity can be cosine similarity, or it can be a degree of closeness calculated based on distance metrics such as Euclidean distance or Mahalanobis distance. This embodiment does not limit the specific form of the similarity. It should be noted that the speaker representation network used here is used to measure the degree of timbre closeness, and its parameters are kept frozen.
[0133] Optionally, the timbre similarity can be determined after the target audio is decoded into a waveform, or it can be determined after decoding the obtained audio latent variables in the intermediate stage of the denoising process. This embodiment does not limit the timing of determining the timbre similarity.
[0134] Then, the timbre similarity is analyzed and compared with the preset similarity threshold. If the timbre similarity is greater than or equal to the preset similarity threshold, there is no need to retrain the audio generation branch. The obtained target audio and target video can be directly output. The preset similarity threshold can be set manually based on experience.
[0135] Step S702: If the timbre similarity is less than the preset similarity threshold, retrain the audio generation branch to obtain the trained audio generation branch.
[0136] If the timbre similarity is less than the preset similarity threshold, that is, the timbre in the target audio is not very similar to the timbre in the reference audio, it means that the inference accuracy of the audio generation branch does not meet the requirements. Therefore, it is necessary to perform a lightweight retraining for the current speaker to obtain a trained audio generation branch. Then, the audio generation branch is regenerated based on the retrained audio generation branch to obtain a result with a timbre that is closer to the reference audio.
[0137] It should be noted that the retraining operation on the audio generation branch is an adaptive process performed during the inference phase on samples from a single speaker. It is not an offline full retraining of the audio-video joint generation model. Therefore, there is no need to prepare large-scale training data again, and it does not affect the existing capabilities of the video generation branch.
[0138] In one feasible approach, the retraining operation of the audio generation branch can be achieved by determining training parameters. In this case, retraining the audio generation branch to obtain the trained audio generation branch may include: determining the training parameters corresponding to the audio generation branch based on timbre similarity, wherein the training parameters include at least the number of iterations, and the number of iterations monotonically decreases as the timbre similarity increases; and retraining the audio generation branch based on the training parameters to obtain the trained audio generation branch.
[0139] After obtaining the timbre similarity, the timbre similarity can be analyzed and processed to determine the training parameters corresponding to the audio generation branch. The training parameters can be determined by analyzing and processing the timbre similarity through a preset mapping relationship. That is, different timbre similarities can determine different training parameters corresponding to the audio generation branch. The obtained training parameters include at least the number of iterations, or the training parameters can also include the learning rate, etc.
[0140] Furthermore, when there are multiple timbre similarities, different training parameters can be generated based on different timbre similarities. Specifically, in the process of generating audio and video of multiple objects to be generated simultaneously using the audio-video joint generation model, multiple target audios can be obtained. Then, each target audio can be determined and analyzed and compared with its corresponding reference audio to obtain the timbre similarity of each of the multiple objects to be generated. Then, the timbre similarity of each of the multiple objects to be generated is normalized and sorted according to the normalized timbre similarity to obtain similarity ranking information.
[0141] Then, based on the similarity ranking information, the first similarity level, the intermediate similarity level, and the second similarity level can be determined. The similarity of the first similarity level is greater than that of the intermediate similarity level, and the similarity of the intermediate similarity level is greater than that of the second similarity level. For the object to be generated with a timbre similarity higher than or at the first similarity level, since the timbre similarity is high, it means that the timbre in the generated target audio is relatively consistent with the timbre in the reference audio, and therefore there is no need to perform a retraining operation on the audio-video joint generation model.
[0142] Similarly, for objects to be generated with timbre similarity in the middle range, training parameters can be determined according to a preset linear mapping relationship based on the values of timbre similarity within the middle range. These training parameters can include an appropriate number of adaptive iterations. For objects to be generated with timbre similarity lower than or in the second similarity range, the training parameters can include a larger number of adaptive iterations. This allows the model to be retrained with a larger number of adaptive iterations.
[0143] Through the above operations, it is possible to determine that for training samples located in the low similarity range, the audio-video joint generation model will undergo more iterative adaptive processing, while for training samples located in the high similarity range, the audio-video joint generation model will not undergo adaptive processing and will directly output the data. For training samples located in the interval between the two, the number of iterations used to retrain the audio-video joint generation model can decrease linearly with the increase of similarity.
[0144] It is worth noting that the stopping method for the model retraining operation may include at least one of the following: stopping after a preset number of iterations; and stopping early when the audio loss function reconstructed by mean square error meets the convergence condition; and / or, sampling at least one time step may include: sampling time steps from all time steps of the inference schedule with a non-fixed step size.
[0145] Furthermore, after obtaining the trained audio generation branch, the method in this embodiment may further include: injecting text description, channel-by-channel modulation amount, and initialized audio noise into the trained audio generation branch to generate adjusted audio, wherein the similarity between the timbre of the adjusted audio and the timbre of the reference audio is higher than the similarity between the timbre of the target audio obtained before retraining and the timbre of the reference audio; and outputting the adjusted audio and the target video. Since retraining only updates the parameters of the audio generation branch and does not change the temporal dimension shape of the audio latent variables, the temporal correspondence between the adjusted audio and the target video remains unchanged.
[0146] Specifically, after the adaptive training of the audio generation branch converges, the audio generation branch can perform standard multi-step denoising inference. That is, the parameters of the audio generation branch updated during the retraining phase only apply to the generation operation of the current speaker, thereby avoiding mutual interference with the generation results of other speakers. As another implementation method, the parameters obtained from retraining can also be cached on a speaker-by-speaker basis so that subsequent generation requests from the same speaker can be directly reused.
[0147] It should be noted that whether to perform the retraining operation of the audio generation branch can be enabled or disabled by the user, or it can be configured to be performed by default for every generation request; in addition, the retraining operation of the audio generation branch can be triggered as a quality fallback measure after the target audio is generated, or it can be triggered in batches according to the similarity classification strategy, and this embodiment does not limit this.
[0148] In this embodiment, by adaptively determining the training parameters of the audio generation branch based on timbre similarity, the computational power of adaptive processing is concentrated on long-tail speakers that are truly difficult to reproduce, while no additional overhead is imposed on samples whose similarity has reached the required level. This improves the timbre reproduction capability while taking into account the overall service efficiency.
[0149] Based on the above embodiments, refer to Figure 8 As shown, retraining the audio generation branch can be achieved not only through training parameters determined by timbre similarity, but also by constructing multiple audio segment pairs using reference audio from the current speaker. In this case, retraining the audio generation branch to obtain the trained audio generation branch can include: Step S801: Segment the reference audio to generate multiple audio segment pairs corresponding to the same speaker. The audio segment pairs include: training audio segments and ground truth audio segments.
[0150] In order to construct sample data for training the audio generation branch based on the current speaker's reference audio, after obtaining the reference audio, the reference audio can be segmented to generate multiple audio segment pairs corresponding to the same speaker. These audio segment pairs include training audio segments and ground truth audio segments.
[0151] Specifically, audio segment pairs can be obtained by performing random or overlapping segmentation on a single reference audio segment provided by the user. All the resulting segments come from the same speaker. Therefore, one segment can be used as a training audio segment to provide timbre conditions, and the other segment can be used as a ground truth audio segment to provide the reconstruction target.
[0152] Furthermore, when retraining the audio generation branch, only a reference audio segment from the speaker can be used as the data source, without collecting additional speech data from the speaker or requiring manual annotation. This achieves single-sample self-supervised adaptive operation on the audio generation branch. Optionally, for audio segment pairs, a minimum duration threshold can be set for the training audio segments. When the duration of a segment in the segmented audio pair is less than the minimum duration threshold, the adaptive processing of the audio generation branch can be skipped, and the output can be directly processed according to the standard inference flow to avoid unstable adaptive performance due to insufficient conditional information.
[0153] Step S802: Retrain the audio generation branch based on multiple audio segment pairs and text descriptions to obtain the trained audio generation branch.
[0154] In one feasible approach, retraining the audio generation branch based on multiple audio segment pairs and text descriptions to obtain the trained audio generation branch may include: using the audio generation branch to perform forward inference on the text description and training audio segments to obtain predicted audio; determining the loss value of the audio loss function based on the difference between the predicted audio and the ground truth audio segments; and using the loss value of the audio loss function to update the parameters of the audio generation branch to obtain the trained audio generation branch.
[0155] The retraining operation of the audio generation branch can be called a single-sample adaptive operation, which is essentially a self-supervised model reconstruction process. Preferably, the audio loss function can be constructed from the reference audio itself without the need to introduce an external scoring network or discriminant network.
[0156] Specifically, after obtaining multiple audio segment pairs, the audio generation branch can be retrained using these pairs and text descriptions. Specifically, several time steps can be sampled from the noise scheduling used in the inference phase. Starting from these sampled time steps, a denoising estimation step is performed to obtain the latent audio variables corresponding to the predicted audio. Based on the result of this denoising estimation step, the mean square error (MSE) between the MSE and the ground truth audio segment (or the latent variable corresponding to the ground truth audio segment) is calculated to reconstruct the audio loss function, which serves as the loss value of the audio loss function. It should be noted that the MSE-based audio loss function is a self-supervised reconstruction objective and does not rely on external scoring or discriminant networks during retraining.
[0157] It is worth noting that during the retraining process of the audio generation branch, only the trainable side parameters in the audio generation branch are updated while the parameters of the video generation branch remain frozen. The trainable side parameters are low-rank incremental parameters connected in parallel with the main parameters in the audio generation branch, while the main parameters remain frozen during single-sample adaptation. Alternatively, the trainable side parameters may include at least one of the following: the projection network parameters corresponding to the channel-wise modulation amount, and the modulation parameters of the normalization layer in the audio generation branch used to scale and translate features along the channel dimension. Alternatively, the trainable side parameters may include parameters of some network layers in the audio generation branch. This embodiment does not limit the range of parameters updated during retraining.
[0158] In another feasible approach, not only is the audio generation branch retrained based on multiple audio segment pairs and text descriptions to obtain a trained audio generation branch, but the video generation branch can also be retrained simultaneously based on a reference graph to obtain a trained video generation branch. In this case, retraining the video generation branch based on the reference graph can include: copying the reference graph to obtain multiple frames; generating a static ground truth video based on the multiple frames; using the video generation branch to perform forward inference on the reference graph, text description, and initialized video noise to obtain a predicted video; obtaining the predicted video and the ground truth video, generating a video loss function, and then using the video loss function to update the parameters of the video generation branch, thereby obtaining the trained video generation branch.
[0159] In this embodiment, the processing method of "segmenting and constructing segment pairs from the same reference audio → calculating the self-supervised reconstruction audio loss function on one-step denoising estimation → updating only the low-rank increment parameters of the audio branch → and outputting the video according to the standard process after adaptive convergence" provides a remedial approach for inference for speakers with poor timbre reproduction. On the one hand, this remediation is based on self-supervised reconstruction and does not introduce external auxiliary networks, resulting in low implementation costs. On the other hand, since the video generation branch is frozen throughout, the adaptive processing will not disturb the image and lip movements of the characters in the generated video, thereby improving the timbre similarity while maintaining the consistency and stability of the image.
[0160] In practical applications, taking the virtual anchor scenario as an example, a content creator wants to generate a video in which a designated real person appears on camera and delivers a narration. The video must match the real person's appearance, voice timbre, and lip movements precisely in time with the spoken lines. This application embodiment provides a method for generating audio and video. The execution entity of this method can be a server, which can provide voice-controlled character video generation services for scenarios such as personalized digital humans, virtual anchors, and film or short video dubbing and translation. Specifically, refer to... Figure 9 As shown, the method for generating audio and video may include the following steps: Step S1: The user submits a request, and the terminal device uploads a reference image, reference audio, and text description.
[0161] Users can upload a reference image of the target person (e.g., an image containing the person's face), reference audio (e.g., a ten-second clip of the person's voice), and a text description (the dialogue to be broadcast and a description of the visual content) to the server via their terminal devices. The reference audio provides a reference timbre, and its actual speech content may differ from the dialogue required by the text description. The server initializes video and audio noise of mutually aligned durations based on a pre-set video generation duration. The text description can be implemented as a prompt word input to the audio-video joint generation model; for example, the text description could be "A person is introducing a new product."
[0162] Step S2: Conditional preprocessing, extracting the timbre from the reference audio and projecting it as a timbre offset.
[0163] The server-side uses a speaker representation network (or speaker verification network or speaker recognition network) that references the frozen audio input parameters. The audio generation branch outputs a global, fixed-length timbre vector, e_spk. For example, the audio generation branch can be a self-supervised speech model such as ECAPA-TDNN, a large-scale speech encoder, or WavLM, and the output timbre vector can be a 1×512 one-dimensional vector. It's important to note that one reference audio corresponds to one timbre vector, not a sequence of frame-by-frame features, meaning that timbre information does not carry word-level content information at the source. Subsequently, the timbre offset projection network (i.e., MLP_s in the above formula, which can be a single-layer linear transformation network or a multilayer perceptron) projects e_spk into a timbre offset of the same dimension as the time-step projection MLP_t(e_t).
[0164] Step S3: Fuse the timbre offset with the time step projection and the learnable basis vector, and project to obtain multiple sets of channel-by-channel modulation quantities.
[0165] For each cascaded double-branch diffusion network block in the audio generation branch (for example, the backbone can contain thirty network blocks, see reference...), Figure 10 As shown, each bi-branch diffuse network block includes at least: a self-attention layer, a feedforward layer, and a normalization layer located before each of them. The six channel-wise modulation quantities required for scaling, translation, and residual gating are obtained by adding the three terms together.
[0166] For example, the timbre offset is MLP_s(e_spk), the time step projection is MLP_t(e_t), specifically the original time step projection based on the diffusion time step in the network; the learnable basis vectors are m^( This is used to represent the differences between different layers (deep and shallow); then the timbre offset MLP_s(e_spk), time step projection MLP_t(e_t), and learnable basis vector m^( The fusion process is performed to obtain the fusion offset "m^( )+MLP_t(e_t)+MLP_s(e_spk)”, and then by adjusting the fusion offset “m^( The mapping operation ")+MLP_t(e_t)+MLP_s(e_spk)" is performed to obtain the following channel-wise modulation quantities: γ_msa, β_msa, α_msa, γ_mlp, β_mlp, α_mlp. Among these, the time step projection and timbre offset can be shared across all network blocks, and the inter-layer differences are only determined by m^( Therefore, during the generation of audio and video, almost no new parameters are added, and the length of the attention sequence is not increased.
[0167] Step S4: Inject the channel-wise modulation into the audio generation branch, denoise in parallel with the video generation branch, and exchange information through cross-modal connections to obtain the denoised audio latent variables and video latent variables.
[0168] Within each network block during the denoising process, intra-block modulation and residual update can be achieved using the following formula: h'i= (1+γ_i)⊙LayerNorm(h_i)+β_i; h{i+1}=h_i+α_i⊙SubLayer_i(h'_i).
[0169] Among them, γ_msa, β_msa, and α_msa act on the normalization layer and residual path before the self-attention layer, γ_mlp, β_mlp, and α_mlp act on the normalization layer and residual path before the feedforward layer, and α is a parameter of the gated residual used to control the degree of access of the sub-layer update amount.
[0170] It is worth noting that the normalization layer itself does not carry learnable affine parameters (i.e., when setting up the network structure, the original learnable affine parameters of LayerNorm are turned off, making the modulation operation the main way to affect the channel statistics). Therefore, the channel-wise scaling and translation that actually reaches the audio word can come from the above (1+γ_i) and β_i - changing the reference timbre directly changes this pair of first-order / second-order statistical factors without affecting the content semantics of the content word. Identity and content are thus decoupled in the model construction, rather than relying on the model to learn them on its own during training.
[0171] In the illustrated embodiment, the channel-wise modulation amount is injected into the audio generation branch but not the video generation branch, and the timbre specification does not change the video-side computation and image quality. During the denoising process, the video generation branch is conditional on a reference image (exemplarily encoded as an image by a video variational autoencoder), video noise, and text description, while the audio generation branch is conditional on text description, the aforementioned channel-wise modulation amount, and audio noise. The two branches exchange information layer by layer through a bidirectional cross-modal connection, and can align the time axes on both sides using rotational position coding, thereby obtaining the denoised audio latent variables and video latent variables, thus ensuring the temporal correspondence between lip movements and speech.
[0172] During the training phase, the timbre vector e_spk is set to zero with a preset probability p (the text description can also be left blank), so that the model can learn to generate under both timbre-condition and timbre-free conditions. For example, p can be 0.1, and its value range can be from 0.05 to 0.15.
[0173] During the inference phase, the three branches mentioned above (i.e., predicting based on three combinations of conditions: empty text and zero timbre, text and zero timbre, and text and timbre) guide the audio generation branch to generate denoised audio latent variables. Different predicted latent variables are obtained under different conditions, using the same video noise and audio noise as input. Specifically, this includes obtaining the first predicted latent variable using an empty text description and a zero timbre vector as conditions. The second latent prediction variable v_text is obtained based on the text description and zero-tone vector, and the third latent prediction variable v_cond is obtained based on the text description and channel-by-channel modulation amount. These are then weighted and summed using the following formula: v= +s_text·(v_text- )+s_spk·(v_cond-v_text); In the above formula, v is the latent variable of the audio after denoising, and s_text and s_spk are manually set weight coefficients. They can be adjusted independently, do not need to meet normalization constraints, and can both be zero: s_text is a preset weight coefficient used to adjust the fit between the dialogue and the spoken content, and s_spk is a preset weight coefficient used to adjust the fidelity of the timbre to the reference audio, thereby decoupling the two guiding forces of "text compliance" and "timbre fidelity", avoiding the situation where the two restrain each other in the conventional two-branch guidance.
[0174] Step S5: Decode the denoised audio and video latent variables to obtain the target audio and target video, and determine the timbre similarity between the target audio and the reference audio.
[0175] After obtaining the denoised audio and video latent variables, they can be decoded using parameter-frozen audio and video decoders respectively, thus obtaining the target audio and target video. Subsequently, the timbre embeddings of the target audio and reference audio are extracted using a parameter-frozen speaker representation network, and the timbre similarity between the target audio and reference audio (e.g., cosine similarity) is calculated. This similarity is used to determine whether subsequent steps need to be performed.
[0176] Step S6: If the timbre similarity is not less than the preset similarity threshold, directly output the target audio and target video; if the timbre similarity is less than the preset similarity threshold, perform a lightweight adaptive training operation on the audio generation branch based on samples for the current speaker.
[0177] Optionally, the server can also normalize the timbre similarity of multiple inference samples and sort them according to the normalization results: samples in the low similarity tier (i.e., the tier corresponding to long-tail difficult speakers) are assigned more iterations, with the number of iterations decreasing monotonically as timbre similarity increases; samples in the high similarity tier do not undergo adaptive training; samples in the middle tier are assigned iterations according to a linear relationship. For example, when the timbre similarity is in the range [0.55, 0.65), the number of iterations can be 8; when the timbre similarity is in the range [0.65, 0.75), the number of iterations can be 4; and when the timbre similarity is not lower than 0.75, retraining can be omitted.
[0178] In addition, whether to enable adaptive training can be determined automatically by the server according to the above rules, or it can be enabled by the user, or it can be configured to always be enabled during deployment.
[0179] For example, during the adaptive training of the audio generation branch, the self-supervised audio reconstruction loss function can be used to update only the low-rank parameters of the audio generation branch, regenerate, and output the audio. Specifically, refer to... Figure 11 As shown, the adaptive training operation may include the following steps: First, the audio segments are randomly segmented from the single reference audio uploaded by the user to construct several audio segment pairs corresponding to the same speaker. Each audio segment pair includes a training audio segment and a ground truth audio segment. When segmenting, a minimum reference duration threshold can be set for the audio segment pairs. When the reference audio is too short to be segmented into effective segment pairs, the process reverts to non-adaptive training and directly adopts the standard inference output.
[0180] Secondly, for each audio segment pair, several time steps (not necessarily equally spaced) are sampled from the noise schedule used in this inference to train the audio segments to propagate forward on the audio generation branch in conjunction with the text description and reference image, and a denoising estimation step is performed to obtain the predicted audio. Specifically, let the current latent variable be z_t and the network prediction speed be... If the current denoising step is t, then proceed according to... =z_t-t· Closed extrapolation yields a one-step denoising prediction operation for clean samples.
[0181] Third, taking the ground truth audio segments in the audio segment pairs that did not participate in the conditional construction as the target, the mean square error (MSE) between the one-step denoising estimation and the audio loss function is calculated to reconstruct the audio loss function, and the audio loss function is backpropagated.
[0182] Fourth, only the bypass parameters (or low-rank adaptive LoRA parameters) of the audio generation branch are updated in parallel, while the video generation branch remains frozen throughout the process. Therefore, the adaptive training operation does not affect the already generated footage.
[0183] It is important to emphasize that the above adaptive training operation is based on self-supervised reconstruction and does not rely on external scoring networks or discriminative models. After adaptive convergence or reaching the configured number of iterations, standard inference is re-executed using the adjusted audio generation branch according to steps S4 to S5 to generate adjusted audio, which is then returned to the terminal device as digital human audio-visual content along with the existing target video. The timbre of the adjusted audio has a higher similarity to the reference audio than the similarity to the target audio before adaptation. Optionally, this adaptive process can be performed in two passes (outputting the video first, then adapting, then outputting the video again) or in a direct pass.
[0184] The audio and video generation method provided in this application embodiment injects the reference timbre into the channel-level statistics of the normalization layer with a channel-by-channel modulation amount, and makes the normalization layer itself not carry learnable affine parameters. The timbre is entered into the generation process through global scaling and translation, so that the timbre of the target audio is truly constrained by the reference audio. In terms of model construction, identity and content are decoupled to avoid content leakage. On the other hand, since no additional lexical units are introduced and only a lightweight projection channel is added, the timbre control capability is obtained with almost zero parameters without increasing the length of the attention sequence, and the memory and computing power overhead is small. Furthermore, through the conditional zeroing during training and the three-branch guidance during inference, the text fit and timbre fidelity can be adjusted independently, taking into account both the accuracy of the dialogue and the timbre reproduction. Moreover, through sample-based self-supervised adaptive inference based on timbre similarity, and by updating only the low-rank parameters of the audio branch and freezing the video branch, the timbre deviation of long-tail speakers can be specifically corrected without affecting the picture. This enables the stable production of audio and video with consistent appearance, controllable timbre, and high alignment of lip movements and speech in scenarios such as personalized digital humans, virtual anchors, and dubbing, effectively expanding the production capacity of digital human content and reducing the cost of live-action filming and post-dubbing.
[0185] Based on the same inventive concept, and referring to Figure 12 As shown, this embodiment provides an audio generation method. The execution subject of this method is an audio generation device, which can be implemented as software or a combination of software and hardware. When the audio generation device is implemented as hardware, it can be various electronic devices capable of performing audio generation operations, including but not limited to software vendors' clients, servers, etc. When the audio generation device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the audio generation method may include: Step S1201: Obtain reference audio and text description.
[0186] Step S1202: Based on the reference audio, determine the channel-by-channel modulation amount corresponding to the timbre in the reference audio. The channel-by-channel modulation amount is used to adjust the channel-level features that characterize the timbre.
[0187] Step S1203: Inject the text description, channel-by-channel modulation amount, and initialized audio noise into the audio generation network to obtain the denoised audio latent variables.
[0188] Step S1204: Decode the audio latent variables to obtain the target audio, the timbre of the target audio being consistent with the timbre of the reference audio.
[0189] In some instances, the audio generation network comprises multiple cascaded bi-branch diffusion network blocks. Determining the channel-wise modulation amount corresponding to the timbre in the reference audio, based on the reference audio, may include: extracting the timbre from the reference audio to obtain a timbre vector; projecting the timbre vector into a timbre offset, the timbre offset having the same dimension as the time-step projection, the time-step projection being determined based on the diffusion time step; fusing the timbre offset, the time-step projection, and the learnable basis vectors of the bi-branch diffusion network blocks to obtain a fused offset; and projecting the fused offset to obtain the channel-wise modulation amount.
[0190] The audio generation method in this embodiment may also include the methods described in the foregoing embodiments. Figures 1-11 The descriptions in the illustrated embodiments are for reference only, and will not be elaborated upon here.
[0191] The audio generation method provided in this embodiment can run only the audio generation network without involving the video generation branch, thus enabling independent speech synthesis with controllable timbre. Therefore, this audio generation method can be applied independently to scenarios that only require audio output, such as dubbing, translation, and voice assistants, expanding the applicability of the timbre injection mechanism and further improving the practicality of the method.
[0192] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.
[0193] refer to Figure 13 As shown, this embodiment provides an audio / video generation apparatus, which is used to perform the above-described... Figure 2 The audio / video generation method shown may include an audio / video generation apparatus that includes: The first acquisition module 11 is used to acquire reference images, reference audio, and text descriptions; The first determining module 12 is used to determine, based on the reference audio, the channel-wise modulation amount corresponding to the timbre in the reference audio, and the channel-wise modulation amount is used to adjust the channel-level features that characterize the timbre; The first injection module 13 is used to inject text description, channel-by-channel modulation amount and initialized audio noise into the audio generation branch to obtain the denoised audio latent variables. The first injection module 13 is also used to inject the reference image, text description and initialized video noise into the video generation branch to obtain the denoised video latent variables. The first processing module 14 is used to decode the audio latent variables and video latent variables to obtain the target audio and the target video, wherein the timbre of the target audio is consistent with the timbre of the reference audio.
[0194] In some instances, the audio generation branch includes multiple cascaded bi-branch diffusion network blocks; when the first determining module 12 determines the channel-wise modulation amount corresponding to the timbre in the reference audio based on the reference audio, the first determining module 12 is used to perform: extracting the timbre from the reference audio to obtain a timbre vector; projecting the timbre vector into a timbre offset, the dimension of the timbre offset being the same as the dimension of the time step projection, the time step projection being determined based on the diffusion time step; fusing the timbre offset, the time step projection, and the learnable basis vectors of the bi-branch diffusion network blocks to obtain a fused offset; and projecting the fused offset to obtain the channel-wise modulation amount.
[0195] In some instances, when the first determining module 12 projects the fusion offset to obtain the channel-wise modulation amount, the first determining module 12 is used to perform: calling the projection network to project the fusion offset to obtain the channel-wise modulation amount, wherein the projection network is used to output multiple sets of channel-wise modulation amounts corresponding to multiple normalization layers in the audio generation branch; the projection network is a single-layer linear transform network or a multilayer perceptron.
[0196] In some instances, the audio generation branch includes multiple cascaded bi-branch diffusion network blocks, each of which includes at least: a self-attention layer, a feedforward layer, a normalization layer preceding the self-attention layer, and a normalization layer preceding the feedforward layer. When the first injection module 13 injects the text description, channel-wise modulation amount, and initialized audio noise into the audio generation branch to obtain the denoised audio latent variables, the first injection module 13 is used to perform: injecting the channel-wise modulation amount into at least one normalization layer in the dual-branch diffusion network block to obtain the normalization layer after injection parameters; injecting the text description, channel-wise modulation amount, and initialized audio noise into the audio generation branch, and using the normalization layer after injection parameters to modulate the timbre features in the denoising process to obtain the denoised audio latent variables.
[0197] In some instances, the channel-by-channel modulation parameters include at least: channel-level scaling parameters, translation parameters, and residual gating parameters. When the first injection module 13 modulates the timbre features in the denoising process using the normalization layer after the injection parameters to obtain the denoised audio latent variables, the first injection module 13 is used to perform: scaling and translating the standard deviation and mean of the features obtained after the normalization operation along the channel dimension using the scaling and translation parameters in the normalization layer after the injection parameters to obtain the layer output features; and gating the update degree corresponding to the layer output features based on the residual gating parameters to obtain the denoised audio latent variables.
[0198] In some instances, when the first injection module 13 injects the text description, channel-wise modulation amount, and initialized audio noise into the audio generation branch to obtain denoised audio latent variables, the first injection module 13 is used to perform the following: inject the text description, channel-wise modulation amount, and initialized audio noise into the audio generation branch; set the channel-wise modulation amount to zero with a preset probability to obtain a zero timbre vector; and set the text description to empty with a preset probability to obtain an empty text description; and use the audio generation branch to generate denoised audio latent variables by using the text description, channel-wise modulation amount, initialized audio noise, zero timbre vector, and empty text description as conditions.
[0199] In some instances, when the first injection module 13 uses the audio generation branch to generate denoised audio latent variables based on the text description, channel-wise modulation amount, initialized audio noise, zero timbre vector, and empty text description, the first injection module 13 performs the following operations: generating a first latent variable based on the audio generation branch using the empty text description, zero timbre vector, and initialized audio noise as conditions; generating a second latent variable based on the audio generation branch using the text description, zero timbre vector, and initialized audio noise as conditions; generating a third latent variable based on the audio generation branch using the text description, channel-wise modulation amount, and initialized audio noise as conditions; and determining the denoised audio latent variables based on the first, second, and third latent variables.
[0200] In some instances, when the first injection module 13 determines the denoised audio latent variables based on the first latent variable, the second latent variable, and the third latent variable, the first injection module 13 is used to perform: determining the first difference information between the second latent variable and the first latent variable, and the second difference information between the third latent variable and the second latent variable; and performing a weighted summation of the first latent variable, the first difference information, and the second difference information to obtain the denoised audio latent variables.
[0201] In some instances, after obtaining the target audio and target video, the first processing module 14 in this embodiment is used to perform: determining the timbre similarity between the target audio and the reference audio; and if the timbre similarity is less than a preset similarity threshold, retraining the audio generation branch to obtain the trained audio generation branch.
[0202] In some instances, when the first processing module 14 retrains the audio generation branch to obtain the trained audio generation branch, the first processing module 14 is used to perform: determining the training parameters corresponding to the audio generation branch based on timbre similarity, wherein the training parameters include at least the number of iterations, and the number of iterations decreases monotonically as the timbre similarity increases; and retraining the audio generation branch based on the training parameters to obtain the trained audio generation branch.
[0203] In some instances, when the first processing module 14 retrains the audio generation branch to obtain the trained audio generation branch, the first processing module 14 is used to perform: segmenting the reference audio to generate multiple audio segment pairs corresponding to the same speaker, the audio segment pairs including: training audio segments and ground truth audio segments; retraining the audio generation branch based on the multiple audio segment pairs and text descriptions to obtain the trained audio generation branch.
[0204] In some instances, when the first processing module 14 retrains the audio generation branch based on multiple audio segment pairs and text descriptions to obtain the trained audio generation branch, the first processing module 14 is used to perform: forward inference on the text description and training audio segments using the audio generation branch to obtain the predicted audio; determine the loss value of the audio loss function based on the difference between the predicted audio and the ground truth audio segments; and update the parameters of the audio generation branch using the loss value of the audio loss function to obtain the trained audio generation branch.
[0205] In some instances, after obtaining the trained audio generation branch, the first processing module 14 in this embodiment is further configured to: inject text description, channel-by-channel modulation amount and initialized audio noise into the trained audio generation branch to generate adjusted audio, wherein the similarity between the timbre of the adjusted audio and the timbre of the reference audio is higher than the similarity between the timbre of the target audio and the timbre of the reference audio; and output the adjusted audio and the target video.
[0206] The audio and video generation device in this embodiment can also perform the aforementioned embodiments. Figures 1-11 The descriptions in the illustrated embodiments are for reference only, and will not be elaborated upon here.
[0207] refer to Figure 14As shown, this embodiment provides an electronic device, which can refer to a complete set of hardware and software infrastructure supporting software operation, data processing, and task execution. It can provide computing resources, storage, communication, and a runtime environment for applications (such as large model inference, scientific simulations, and web services). This electronic device is used to perform the above-described tasks. Figure 2 The audio and video generation method shown may, in practice, include a memory 24 and a processor 25 in the electronic device.
[0208] Memory 24 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.
[0209] Processor 25, coupled to memory 24, executes a computer program in memory 24 for: acquiring a reference image, reference audio, and text description; determining a channel-wise modulation amount corresponding to the timbre in the reference audio based on the reference audio, the channel-wise modulation amount being used to adjust the channel-level features characterizing the timbre; injecting the text description, channel-wise modulation amount, and initialized audio noise into the audio generation branch to obtain denoised audio latent variables; injecting the reference image, text description, and initialized video noise into the video generation branch to obtain denoised video latent variables; and decoding the audio latent variables and video latent variables to obtain target audio and target video, the timbre of the target audio being consistent with the timbre of the reference audio.
[0210] Further, refer to Figure 14 As shown, the electronic device also includes other components such as a communication component 26, a display 27, a power supply component 28, and an audio component 29. Figure 14 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 14 The components shown. Additionally... Figure 14 The components within the center frame are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, smartphone, or Internet of Things (IoT) device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a desktop computer, laptop computer, or smartphone, it may include... Figure 14 The components within the center frame; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, it may not include... Figure 14 The component within the center frame.
[0211] refer to Figure 15 As shown, this embodiment provides an audio generation apparatus, which is used to perform the above-described... Figure 12 The method for generating audio shown may include an audio generating apparatus that includes: The second acquisition module 31 is used to acquire reference audio and text description; The second determining module 32 is used to determine the channel-by-channel modulation amount corresponding to the timbre in the reference audio based on the reference audio. The channel-by-channel modulation amount is used to adjust the channel-level features that characterize the timbre. The second injection module 33 is used to inject text description, channel-by-channel modulation amount and initialized audio noise into the audio generation network to obtain the denoised audio latent variables. The second processing module 34 is used to decode the audio latent variables to obtain the target audio, the timbre of which is consistent with that of the reference audio.
[0212] In some instances, the audio generation network includes multiple cascaded bi-branch diffusion network blocks; when the second determining module 32 determines the channel-wise modulation amount corresponding to the timbre in the reference audio based on the reference audio, the second determining module 32 is used to perform: extracting the timbre from the reference audio to obtain a timbre vector; projecting the timbre vector into a timbre offset, the dimension of which is the same as the dimension of the time step projection, the time step projection being determined based on the diffusion time step; fusing the timbre offset, the time step projection, and the learnable basis vectors of the bi-branch diffusion network blocks to obtain a fused offset; and projecting the fused offset to obtain the channel-wise modulation amount.
[0213] The audio generation apparatus in this embodiment can also perform the aforementioned embodiments. Figure 12 The descriptions in the illustrated embodiments are for reference only, and will not be elaborated upon here.
[0214] refer to Figure 16 As shown, this embodiment provides an electronic device, which can refer to a complete set of hardware and software infrastructure supporting software operation, data processing, and task execution. It can provide computing resources, storage, communication, and a runtime environment for applications (such as large model inference, scientific simulations, and web services). This electronic device is used to perform the above-described tasks. Figure 12 The audio generation method shown may, in practice, include a memory 44 and a processor 45 in the electronic device.
[0215] Memory 44 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.
[0216] The processor 45, coupled to the memory 44, executes a computer program in the memory 44 for: acquiring a reference audio and a text description; determining a channel-wise modulation amount corresponding to the timbre in the reference audio based on the reference audio, the channel-wise modulation amount being used to adjust the channel-level features characterizing the timbre; injecting the text description, the channel-wise modulation amount, and initialized audio noise into an audio generation network to obtain denoised audio latent variables; and decoding the audio latent variables to obtain a target audio, the timbre of which is consistent with the timbre of the reference audio.
[0217] Further, refer to Figure 16 As shown, the electronic device also includes other components such as a communication component 46, a display 47, a power supply component 48, and an audio component 49. Figure 16 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 16 The components shown. Additionally... Figure 16 The components within the center frame are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, smartphone, or Internet of Things (IoT) device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a desktop computer, laptop computer, or smartphone, it may include... Figure 16 The components within the center frame; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, it may not include... Figure 16 The component within the center frame.
[0218] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0219] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.
[0220] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.
[0221] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.
[0222] The aforementioned audio component can be configured to output and input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.
[0223] Accordingly, embodiments of this application also provide a computer storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.
[0224] Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device can be implemented as a means to implement the corresponding functions in the above method embodiments.
[0225] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0226] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for generating audio and video, characterized in that, include: Obtain reference images, reference audio, and text descriptions; Based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio is determined, and the channel-wise modulation amount is used to adjust the channel-level features that characterize the timbre; The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation branch to obtain the denoised audio latent variables; The reference image, the text description, and the initialized video noise are injected into the video generation branch to obtain the denoised video latent variables. The audio latent variables and the video latent variables are decoded to obtain the target audio and the target video, wherein the timbre of the target audio is consistent with the timbre of the reference audio.
2. The method according to claim 1, characterized in that, The audio generation branch includes multiple cascaded dual-branch diffusion network blocks; based on the reference audio, determining the channel-by-channel modulation amount corresponding to the timbre in the reference audio includes: The timbre is extracted from the reference audio to obtain a timbre vector; The timbre vector is projected as a timbre offset, the dimension of which is the same as the dimension of the time step projection, and the time step projection is determined based on the diffusion time step; The timbre offset, the time step projection, and the learnable basis vectors of the dual-branch diffusion network block are fused to obtain the fused offset. The channel-by-channel modulation amount is obtained by projecting the fusion offset.
3. The method according to claim 2, characterized in that, Projecting the fusion offset to obtain the channel-by-channel modulation amount includes: The projection network is invoked to project the fusion offset to obtain the channel-wise modulation amount, wherein the projection network is used to output multiple sets of the channel-wise modulation amounts corresponding to multiple normalization layers in the audio generation branch; the projection network is a single-layer linear transform network or a multilayer perceptron.
4. The method according to claim 1, characterized in that, The audio generation branch includes multiple cascaded bi-branch diffusion network blocks, and each bi-branch diffusion network block includes at least: a self-attention layer, a feedforward layer, a normalization layer located before the self-attention layer, and a normalization layer located before the feedforward layer. The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation branch to obtain the denoised audio latent variables, including: The channel-by-channel modulation amount is injected into at least one of the normalized layers in the dual-branch diffusion network block to obtain the normalized layer after injection parameters; The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation branch, and the timbre features in the denoising process are modulated using the normalization layer after the injection parameters to obtain the denoised audio latent variables.
5. The method according to claim 4, characterized in that, The channel-by-channel modulation amount includes at least: channel-level scaling parameters, translation parameters, and residual gating parameters; the normalization layer after the injected parameters is used to modulate the timbre features during the denoising process to obtain the denoised audio latent variables, including: The scaling and translation parameters in the normalized layer after the injection parameters are used to scale and translate the standard deviation and mean of the features obtained after the normalization operation along the channel dimension to obtain the layer output features; Based on the residual gating parameters, the update degree corresponding to the layer output features is gated to obtain the denoised audio latent variables.
6. The method according to claim 1, characterized in that, The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation branch to obtain the denoised audio latent variables, including: Determine the zero timbre vector corresponding to the channel-by-channel modulation amount and the empty text description corresponding to the text description. The zero timbre vector is obtained by setting the timbre offset of the channel-by-channel modulation amount to zero, and the empty text description is obtained by setting the text description to empty. Using the audio generation branch, the text description, the channel-by-channel modulation amount, the initialized audio noise, the zero timbre vector, and the empty text description are used as conditions to generate denoised audio latent variables.
7. The method according to claim 6, characterized in that, Using the audio generation branch, the text description, the channel-by-channel modulation amount, the initialized audio noise, the zero-timbre vector, and the empty text description as conditions, denoised audio latent variables are generated, including: Using the audio generation branch, the empty text description, the zero timbre vector, and the initialized audio noise are used as conditions to generate the first latent variable; Using the audio generation branch, the text description, the zero timbre vector, and the initialized audio noise are used as conditions to generate a second latent variable; Using the audio generation branch, the text description, the channel-by-channel modulation amount, and the initialized audio noise are used as conditions to generate a third latent variable; Based on the first latent variable, the second latent variable, and the third latent variable, the denoised audio latent variables are determined.
8. The method according to claim 7, characterized in that, Based on the first latent variable, the second latent variable, and the third latent variable, the denoised audio latent variables are determined, including: Determine the first difference information between the second latent variable and the first latent variable, and the second difference information between the third latent variable and the second latent variable; The first latent variable, the first difference information, and the second difference information are weighted and summed to obtain the denoised audio latent variable.
9. The method according to any one of claims 1-8, characterized in that, After obtaining the target audio and target video, the method further includes: Determine the timbre similarity between the target audio and the reference audio; If the timbre similarity is less than a preset similarity threshold, the audio generation branch is retrained to obtain a trained audio generation branch.
10. The method according to claim 9, characterized in that, The audio generation branch is retrained to obtain a trained audio generation branch, including: Based on the timbre similarity, the training parameters corresponding to the audio generation branch are determined. The training parameters include at least the number of iterations, wherein the number of iterations decreases monotonically as the timbre similarity increases. The audio generation branch is retrained based on the training parameters to obtain the trained audio generation branch.
11. The method according to claim 9, characterized in that, The audio generation branch is retrained to obtain a trained audio generation branch, including: The reference audio is segmented to generate multiple audio segment pairs corresponding to the same speaker. The audio segment pairs include: training audio segments and ground truth audio segments. The audio generation branch is retrained based on the multiple audio segment pairs and the text description to obtain the trained audio generation branch.
12. The method according to claim 11, characterized in that, The audio generation branch is retrained based on the multiple audio segment pairs and the text description to obtain a trained audio generation branch, including: The audio generation branch is used to perform forward inference on the text description and the training audio segment to obtain the predicted audio. Based on the difference between the predicted audio and the ground truth audio segment, the loss value of the audio loss function is determined; The audio generation branch is updated with parameters using the loss value of the audio loss function to obtain the trained audio generation branch.
13. The method according to claim 9, characterized in that, After obtaining the trained audio generation branch, the method further includes: The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the trained audio generation branch to generate the adjusted audio, wherein the timbre of the adjusted audio is more similar to the timbre of the reference audio than the timbre of the target audio is similar to the timbre of the reference audio. Output the adjusted audio and the target video.
14. A method for generating audio, characterized in that, include: Obtain reference audio and text description; Based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio is determined, and the channel-wise modulation amount is used to adjust the channel-level features that characterize the timbre; The text description, the channel-by-channel modulation amount, and the initialized audio noise are injected into the audio generation network to obtain the denoised audio latent variables; The audio latent variables are decoded to obtain the target audio, the timbre of which is consistent with that of the reference audio.
15. The method according to claim 14, characterized in that, The audio generation network includes multiple cascaded dual-branch diffusion network blocks; based on the reference audio, determining the channel-wise modulation amount corresponding to the timbre in the reference audio includes: The timbre is extracted from the reference audio to obtain a timbre vector; The timbre vector is projected as a timbre offset, the dimension of which is the same as the dimension of the time step projection, and the time step projection is determined based on the diffusion time step; The timbre offset, the time step projection, and the learnable basis vectors of the dual-branch diffusion network block are fused to obtain the fused offset. The channel-by-channel modulation amount is obtained by projecting the fusion offset.
16. An audio / video generation apparatus, characterized in that, include: The first acquisition module is used to acquire reference images, reference audio, and text descriptions; The first determining module is used to determine, based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio, wherein the channel-wise modulation amount is used to adjust the channel-level features characterizing the timbre; The first injection module is used to inject the text description, the channel-by-channel modulation amount, and the initialized audio noise into the audio generation branch to obtain the denoised audio latent variables. The first injection module is further configured to inject the reference image, the text description, and the initialized video noise into the video generation branch to obtain the denoised video latent variables; The first processing module is used to decode the audio latent variable and the video latent variable to obtain the target audio and the target video, wherein the timbre of the target audio is consistent with the timbre of the reference audio.
17. An audio generation apparatus, characterized in that, include: The second acquisition module is used to acquire reference audio and text description; The second determining module is used to determine, based on the reference audio, a channel-wise modulation amount corresponding to the timbre in the reference audio, wherein the channel-wise modulation amount is used to adjust the channel-level features characterizing the timbre; The second injection module is used to inject the text description, the channel-by-channel modulation amount, and the initialized audio noise into the audio generation network to obtain the denoised audio latent variables. The second processing module is used to decode the audio latent variables to obtain the target audio, the timbre of which is consistent with the timbre of the reference audio.
18. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method of any one of claims 1-15.
19. A computer storage medium, characterized in that, Used to store a computer program that, when executed by a computer, implements the method of any one of claims 1-15.
20. A computer program product, characterized in that, include: A computer program or instruction that, when executed by one or more processors, causes the one or more processors to perform the steps of the method according to any one of claims 1-15.
