Concomitant audio generation method, related apparatus, and medium

Abstract

Provided in the present disclosure are a concomitant audio generation method, a related apparatus, and a medium. The method comprises: extracting a visual feature of a target video, and acquiring a target sound event semantic feature of the target video; inputting the visual feature into a diffusion model, which comprises a coding layer used for performing, on the basis of the visual feature, diffusion processing on an audio feature to be subjected to diffusion; and on the basis of the target sound event semantic feature, generating a first adjustment feature corresponding to the coding layer, adjusting a first output of the coding layer on the basis of the first adjustment feature so as to obtain a second output, and generating a concomitant audio of the target video on the basis of the second output, such that the concomitant audio comprises an audio clip corresponding to the target sound event semantic feature. The present disclosure enables an audio, which is generated on the basis of a video, to comprise audio clips corresponding to all sound events in the video, thereby generating audio that can achieve accurately semantically align with the video. The present disclosure can be applied to scenarios such as automatic dubbing and text-to-video dubbing.

Description

Accompanying audio generation methods, related devices and media

[0001] This application claims priority to Chinese Patent Application No. 202411217392.7, filed on September 2, 2024, entitled “Method, Apparatus and Medium for Generating Accompanying Audio”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of audio generation technology, and more specifically to the generation of accompanying audio. Background Technology

[0003] Voice-over is a crucial part of the video creation process. Traditional voice-over requires a significant amount of time and effort from video creators. This is especially true in the creation of self-media videos, where video footage can be created with the assistance of numerous video creation tools, or even automatically generated by artificial intelligence models. However, videos generated by AI models are often silent, requiring creators to add voice-over, resulting in a substantial portion of their time and energy being spent on voice-over.

[0004] In existing technologies, there are methods that automatically generate accompanying audio for videos using artificial intelligence models. These methods typically encode the video into corresponding video features, then input these features into an audio generation model to generate the corresponding audio. However, the audio generation model struggles to capture all the semantics contained in the video features, resulting in semantic misalignment between the generated audio and the video. Summary of the Invention

[0005] This disclosure provides an accompanying audio generation method, related apparatus, and medium that enables audio generated from video to be semantically and accurately aligned with the video.

[0006] According to one aspect of this disclosure, a method for generating accompanying audio is provided, comprising:

[0007] Extract the visual features of the target video and obtain the semantic features of the target sound events in the target video;

[0008] The visual features are input into a diffusion model, which includes an encoding layer for performing diffusion processing on the audio features to be diffused based on the visual features.

[0009] Based on the semantic features of the target sound event, a first adjustment feature corresponding to the coding layer is generated. Based on the first adjustment feature, the first output of the coding layer is adjusted to obtain a second output. Based on the second output, the accompanying audio of the target video is generated so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event.

[0010] According to one aspect of this disclosure, an accompanying audio generation apparatus is provided, comprising:

[0011] The feature extraction unit is used to extract the visual features of the target video and obtain the semantic features of the target sound events in the target video.

[0012] A diffusion execution unit is used to input the visual features into a diffusion model, the diffusion model including an encoding layer for performing diffusion processing on the audio features to be diffused according to the visual features;

[0013] The diffusion adjustment unit is used to generate a first adjustment feature corresponding to the coding layer based on the semantic features of the target sound event, adjust the first output of the coding layer based on the first adjustment feature to obtain a second output, and generate the accompanying audio of the target video based on the second output, so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event.

[0014] This disclosure not only relies on the visual features of the target video to generate accompanying audio, but also acquires the semantic features of the target sound events in the target video. The semantic features of the target sound events reflect the sound events present in the target video that can produce sound effects. The visual features are input into a diffusion model, which includes an encoding layer capable of diffusion processing the audio features to be diffused based on the visual features. A first adjustment feature of the encoding layer is generated based on the semantic features of the target sound events. This first adjustment feature can fine-tune the intermediate data during the diffusion process, thereby adjusting the first output of the encoding layer to obtain a second output. The accompanying audio of the target video can be accurately generated based on the second output. Thus, the information reflected by the semantic features of the target sound events is injected into the diffusion process, ensuring that the diffusion model can capture the sound events in the target video during diffusion processing. Since the semantics of each sound event in the target video are reflected in the first adjustment feature, which is used to adjust the first output of the encoding layer, the accompanying audio finally generated by the diffusion model can include audio segments corresponding to all target sound events in the target video. Therefore, the final generated accompanying audio is semantically aligned with the target video and will not miss any sound events in the video. Attached Figure Description

[0015] Figure 1 is a system architecture diagram of the accompanying audio generation method according to an embodiment of the present disclosure;

[0016] Figures 2A-2E are application scenario interface diagrams of the embodiments of this disclosure applied in the application scenario of accompanying audio generation, and schematic diagrams comparing them with conventional methods in related technologies;

[0017] Figure 3 is a flowchart of an accompanying audio generation method according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of a process for generating accompanying audio based on a target video according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of the architecture of a diffusion model according to an embodiment of the present disclosure;

[0020] Figure 6 is a flowchart of step 320 in Figure 3;

[0021] Figure 7 is a flowchart of step 320 in Figure 3;

[0022] Figure 8 is a schematic diagram of the input and output of the coding layer according to an embodiment of the present disclosure;

[0023] Figure 9 is a flowchart of step 310 in Figure 3;

[0024] Figure 10 is a flowchart of step 310 in Figure 3;

[0025] Figure 11 is a flowchart of updating the semantic features of a target sound event based on positive and negative prompt words according to an embodiment of the present disclosure;

[0026] Figure 12 is a schematic diagram of an interface for obtaining positive prompts and negative prompts according to an embodiment of the present disclosure;

[0027] Figure 13 is a flowchart of step 330 in Figure 3;

[0028] Figure 14 is a schematic flowchart of a process for generating a first adjustment feature based on the semantic features of a target sound event and adjusting the first output of the coding layer according to an embodiment of the present disclosure.

[0029] Figure 15 is a flowchart of a first adjustment feature being weighted based on a weighting coefficient and then superimposed onto a first output according to an embodiment of the present disclosure;

[0030] Figure 16 is a schematic diagram of the interface for obtaining weight coefficients according to an embodiment of the present disclosure;

[0031] Figure 17 is a flowchart of adjusting the diffusion processing based on sound event timestamp features according to an embodiment of the present disclosure;

[0032] Figure 18 is a comparative schematic diagram of the diffusion processing before and after adjusting the diffusion process based on the timestamp features of sound events, according to an embodiment of the present disclosure.

[0033] Figure 19 is a flowchart of step 1710 in Figure 17;

[0034] Figure 20 is a flowchart of determining whether to acquire the timestamp feature of a sound event according to an embodiment of the present disclosure;

[0035] Figure 21 is a flowchart of step 1710 in Figure 17;

[0036] Figure 22 is a schematic diagram of an interface for marking sound event timestamps according to an embodiment of the present disclosure;

[0037] Figure 23 is a flowchart of step 1710 in Figure 17;

[0038] Figure 24 is a schematic diagram of an interface for automatically detecting sound event timestamps according to an embodiment of the present disclosure;

[0039] Figure 25 is a flowchart of step 1730 in Figure 17;

[0040] Figure 26 is a flowchart of training a semantic feature generation model according to an embodiment of the present disclosure;

[0041] Figure 27 is a flowchart of training a semantic feature adapter according to an embodiment of the present disclosure;

[0042] Figure 28 is a schematic diagram of the structure of a semantic feature adapter according to an embodiment of the present disclosure;

[0043] Figure 29 is a schematic diagram of the process of training a semantic feature adapter according to an embodiment of the present disclosure;

[0044] Figure 30 is a flowchart of training a visual encoder and an audio encoder according to an embodiment of the present disclosure;

[0045] Figure 31 is a process diagram of a specific implementation example of the accompanying audio generation method according to an embodiment of the present disclosure;

[0046] Figure 32 is a block diagram of an accompanying audio generation apparatus according to an embodiment of the present disclosure;

[0047] Figure 33 is a structural diagram of an object terminal according to an embodiment of the present disclosure;

[0048] Figure 34 is a server structure diagram according to an embodiment of the present disclosure. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0050] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0051] Multimodal models are artificial intelligence models capable of processing cross-modal data. They learn the relationships between data from different modalities, understanding input information across modalities. The multimodal input information is then transformed and fused into a feature representation that the model can understand. Based on this feature representation, the model performs corresponding tasks and obtains the appropriate output. In related technologies, multimodal models are commonly found in scenarios such as text-to-image processing, text-to-video processing, and multimodal search.

[0052] A visual language model (VLM) is a multimodal model used to process visual and textual information. It learns the relationships between image and text information to perform corresponding tasks based on input images and text, and generates corresponding text input. It can handle various cross-modal data processing tasks such as visual question answering, image captioning, text understanding, and image recognition. It can receive image or text input, understand the semantic information within the input, and generate corresponding text output.

[0053] Diffusion model: A generative model that can be used for cross-modal data generation tasks, such as text-to-image, text-to-video, and video-to-speech. Inspired by non-equilibrium thermodynamics, the diffusion model defines a Markov chain of diffusion steps to slowly add random noise to the data, transforming the original data into noisy data. Then, by learning to reverse the diffusion process, it learns how to construct the desired generative model from the noise. The training process of the diffusion model includes forward diffusion and backward diffusion. During training, the diffusion model first receives original data of one modality A and diffusion guidance data of another modality corresponding to the original data. The forward diffusion process includes multiple noise-adding steps. In each noise-adding step, the diffusion model predicts noise and adds it to the original data of the diffusion model. By gradually adding noise through multiple noise-adding steps, the original data is transformed into noise data composed entirely of noise. At the same time, the diffusion model records the noise added in each noise-adding step. After converting the original input into noise data, multiple denoising steps are performed on the noise data. These denoising steps correspond one-to-one with the noise-adding steps. In each denoising step, a noise is predicted based on the diffusion guidance data and the noise data of the current denoising step, and the predicted noise is subtracted from the noise data. By comparing the noise predicted in the denoising step with the noise added to the original data in the corresponding noise-adding step, the loss value is calculated. Thus, the diffusion model learns how to predict the corresponding noise based on the data of one modality, thereby generating data for another modality. After training, when the diffusion model is actually used, the reverse diffusion process of the diffusion model is used to predict the noise corresponding to each denoising step based on the given input, and then the data composed of noise is denoised to generate the corresponding output.

[0054] Text-generated video: A technology that automatically generates videos using an artificial intelligence model. It works by inputting text descriptions into a trained multimodal model, which learns during training how to capture the semantic information of the text descriptions and uses this information to generate a series of consecutive video frames, thus producing the corresponding video.

[0055] Video is an important form of content presentation, and video dubbing is a crucial part of the video creation process. By adding dubbing to videos, the visuals and audio can be combined, allowing for a more intuitive transmission of information. However, traditional video dubbing methods require a significant amount of time and effort from video creators. In particular, with the development of text-based video technology, video creators can automatically generate videos using AI models. However, videos generated by AI models are often silent, requiring creators to spend considerable time and effort dubbing them.

[0056] In related technologies, there are methods for automatically adding audio to silent videos using artificial intelligence models. These methods often encode the video into corresponding video features and input these features into an audio generation model, so that the audio generation model can generate corresponding audio based on the video features. However, video features are used to represent the visual information of video images, and the correspondence between visual images and sound events is relatively complex. The audio generation model has difficulty capturing all the sound events that occur in the video from these video features that represent visual information, resulting in the generated audio lacking the sound effects corresponding to some sound events, making the audio generated by the model semantically misaligned with the video.

[0057] Based on this, this disclosure proposes an accompanying audio generation method, related apparatus, and medium, which enables the accompanying audio generated based on the target video to contain audio segments corresponding to each sound event in the target video, and the generated accompanying audio can be accurately aligned semantically with the target video.

[0058] System architecture and scenario descriptions applied in the embodiments of this disclosure:

[0059] Figure 1 is a system architecture diagram of the accompanying audio generation method according to an embodiment of the present disclosure. It includes: a target terminal 110, an Internet 120, a gateway 130, and a server 140.

[0060] The object terminal 110 is a device used by the object to upload target videos, enabling the server 140 to generate corresponding audio based on the target videos. It includes various forms such as desktop computers, laptops, PDAs (Personal Digital Assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, forming a single object terminal. The object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0061] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0062] Server 140 refers to a computer system capable of providing audio generation services to target terminal 110. Compared to target terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0063] The embodiments disclosed herein can be applied to various scenarios, such as the automatic dubbing scenarios shown in Figures 2A to 2E.

[0064] Referring to Figure 2A, in related technologies, the dubbing of silent videos is often done manually by voice actors using appropriate dubbing tools, which requires a lot of time and effort from the voice actors.

[0065] In this disclosure, referring to Figure 2B, the object opens the audio generation software through the object terminal 110. The audio generation software interface displays a prompt to upload a video file, as well as an upload box for uploading video files. The object can click the upload box in the software interface and then select the video file to be used to generate audio from the local files of the terminal (i.e., the local files of the object terminal 110), or drag and drop the video file from the local files of the terminal into the upload box, thereby uploading the video file to be dubbed to the audio generation software.

[0066] Referring to Figure 2C, the object uploaded a 60-second video file, and corresponding accompanying audio needs to be generated. After receiving the video file uploaded by the object, the audio generation software will display a video file playback window, an option window for setting some control parameters of the accompanying audio method disclosed in this disclosure during execution, a preview window for the subsequently generated accompanying audio, and a preview window for the video obtained after the generated accompanying audio is synthesized into the target video.

[0067] Subsequently, the user sets some control parameters for the audio generation process in the interface of the audio generation software. As shown in Figure 2D, in the semantic alignment parameter setting window, automatic generation is selected, causing server 140 to automatically call the artificial intelligence model to generate video description text describing the sound events in the video file. The user can also set the semantic alignment weight coefficient and batch size. The batch size represents the number of accompanying audio files to be generated. As shown in Figure 2D, the weight coefficient is set to 1, and the batch size is set to 4. After setting the corresponding control parameters in the audio generation software interface, as shown in Figure 2E, the user clicks the "Generate Audio" option. The audio generation software executes the accompanying audio generation method of this disclosure to generate accompanying audio corresponding to the input video file and displays four candidate accompanying audio files in the audio generation result window for the user to select. The user can select these candidate accompanying audio files to preview the composite result after merging the selected candidate accompanying audio files into the video in the composite video result window, and then select the final accompanying audio file from the candidate accompanying audio files.

[0068] It should be understood that the above description only illustrates some application scenarios of this disclosure. The business scenarios to which this disclosure can be applied may include, but are not limited to, the specific embodiments described above. For example, in the scenario of text-based video, corresponding accompanying audio can be generated based on the video generated by the text-based video application, and then the video with synthesized accompanying audio can be output.

[0069] General description of the embodiments disclosed herein:

[0070] The accompanying audio generation method of this disclosure embodiment can be executed by a computer device, which is a target terminal and / or a server. That is, the accompanying audio generation method of this disclosure embodiment can be executed on the target terminal 110, or on the server 140, or partly on the target terminal 110 and partly on the server 140.

[0071] As shown in Figure 3, according to an embodiment of this disclosure, a method for generating accompanying audio is proposed. The method for generating accompanying audio includes:

[0072] Step 310: Extract the visual features of the target video and obtain the semantic features of the target sound events in the target video;

[0073] Step 320: Input the visual features into the diffusion model, which includes an encoding layer for diffusion processing of the audio features to be diffused based on the visual features;

[0074] Step 330: Generate a first adjustment feature corresponding to the coding layer based on the semantic features of the target sound event; adjust the first output of the coding layer based on the first adjustment feature to obtain a second output; generate the accompanying audio of the target video based on the second output, so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event.

[0075] In step 310, the target video refers to the video for which corresponding accompanying audio needs to be generated. This can be understood as a silent video. For example, the target video could be a model generated by the object through inputting a text description into a text-based video model, or it could be a video created by the object using other video production tools, such as animations created by animators.

[0076] Visual features are features used to characterize the visual information of video frames in a target video. Referring to Figure 4, visual features can be obtained by encoding the target video using a trained visual encoder. Visual features can be a sequence of data or a multidimensional tensor. Specifically, in the visual encoder, the target video is first converted into multiple video frames. Then, features of these video frames are extracted using a convolutional neural network, and finally, these features are fused through a fully connected layer to obtain the visual features corresponding to the target video. The training process of the visual encoder will be explained later and will not be elaborated here.

[0077] Target sound event semantic features are semantic features used to characterize target sound events present in a target video. A target sound event refers to an event contained in the video frame of the target video that produces a corresponding sound effect. For example, if there is a scene of a dog barking in the target video, then after detecting the target video, it can be determined that there is a target sound event of "dog barking". Specifically, target sound event semantic features can be obtained by encoding a semantic description of the sound event. The semantic description of the sound event is text used to describe the sound event in the target video. For example, when the target sound event of "dog barking" is detected in the target video, the corresponding semantic description of the sound event is generated: "The sound events present in the video include: dog barking". Then, this semantic description of the sound event is encoded by a text modality encoder to obtain the corresponding target sound event semantic features. Specifically, target sound events in the target video can be identified through a visual language model, and the corresponding semantic description of the sound event can be generated. Then, the semantic description of the sound event is encoded by a text modality encoder to obtain the target sound event semantic features.

[0078] The details of step 310 will be described in detail later, and will not be repeated here.

[0079] In step 320, the diffusion model is a pre-trained model used to predict noise based on visual features, thereby generating the corresponding audio.

[0080] The encoding layer is a functional layer composed of neural network layers in the backbone network of a diffusion model. For example, referring to Figure 5, the diffusion model is based on the Unet network architecture. The Unet network architecture diffusion model includes multiple one-to-one encoders and decoders, each encoder being an encoding layer and each decoder being a decoding layer. Furthermore, referring to Figure 5, in the Unet architecture diffusion model, multiple encoders are cascaded sequentially, and multiple decoders are also cascaded sequentially. An intermediate connection layer is set between the multiple encoders and multiple decoders; this intermediate connection layer can also be regarded as an encoding layer.

[0081] In another embodiment, the diffusion model can also be a model based on a diffusion transformer (DiT) architecture. The diffusion model of the diffusion transformer architecture includes a neural network layer for preprocessing the model input, multiple diffusion transformer blocks (DiT blocks), and a linear layer for decoding the final output of the multiple diffusion transformer blocks. The multiple diffusion transformer blocks are cascaded in sequence, and the output of each diffusion transformer block is used as the input of the next diffusion transformer block. Each diffusion transformer block can be regarded as an encoding layer.

[0082] The audio feature to be diffused is the original audio feature before the diffusion model performs diffusion processing. It can be understood that the diffusion processing involves the diffusion model performing multiple denoising steps, predicting corresponding noise at each step, thus denoising a given set of noisy data to generate the corresponding output. The audio feature to be diffused can be an audio feature composed entirely of random noise, for example, an audio feature composed of Gaussian noise. Specifically, since the audio feature to be diffused is a feature composed of random noise, it is independent of the input of the diffusion model and can be automatically generated by the diffusion model before performing diffusion processing; for example, it can be generated by the diffusion model through random sampling based on a Gaussian distribution.

[0083] The process of using visual features to denoise audio features involves using visual features as guiding vectors to instruct the diffusion model to predict the corresponding noise. In essence, the diffusion model is a cross-modal generative model. During the diffusion process, it predicts the noise corresponding to each denoising step based on the input visual features and subtracts the predicted noise from the initially generated audio features to be diffused, thus performing multi-step denoising to generate audio data corresponding to the visual features.

[0084] In one embodiment, in the diffusion model of the Unet network architecture, each encoder includes a cross-attention mechanism. Referring to Figure 6, step 320 involves diffusion processing of the audio features to be diffused based on visual features, including:

[0085] Step 610: In the coding layer, determine the intermediate features of the audio to be diffused based on the features of the audio to be diffused;

[0086] Step 620: Generate a query matrix based on the intermediate features of the audio to be diffused, and generate a key matrix and a value matrix based on the visual features;

[0087] Step 630: Perform cross-attention processing based on the query matrix, key matrix, and value matrix, and determine the first output corresponding to the encoding layer based on the result of the cross-attention processing.

[0088] In step 610, the intermediate audio feature to be diffused refers to the intermediate feature representation obtained after processing the audio feature to be diffused by a portion of the neural network layers of the diffusion model, and then input into the multi-head attention mechanism in the coding layer. It can be understood that in the diffusion model, each coding layer includes other neural network layers besides the multi-head attention mechanism, such as residual blocks. After the audio feature to be diffused is input into the coding layer, it first passes through these neural network layers within the coding layer and undergoes corresponding processing to obtain the corresponding intermediate feature representation, i.e., the intermediate audio feature to be diffused, before being input into the multi-head attention mechanism within the coding layer. Furthermore, multiple coding layers are cascaded; the audio feature to be diffused passes through each coding layer sequentially. Except for the first coding layer, the intermediate feature representation received by each of the other coding layers is actually obtained after the previous coding layer has performed corresponding processing on the audio feature to be diffused. Based on this, in this embodiment, in the coding layer, the intermediate audio feature to be diffused corresponding to the audio feature to be diffused is first determined, and this intermediate audio feature to be diffused is used as the input to the multi-head attention mechanism in the coding layer.

[0089] In step 620, a query matrix is ​​determined based on the intermediate features of the audio to be diffused, and a key matrix and a value matrix are determined based on the visual features. Specifically, in a multi-head attention mechanism, three weight matrices are included: a query weight matrix, a key weight matrix, and a value weight matrix. These three weight matrices are determined during the training of the diffusion model and will not be elaborated here. The intermediate features of the audio to be diffused are multiplied by the query weight matrix, and the visual features are multiplied by the key weight matrix and the value weight matrix, respectively, thus obtaining the query matrix, key matrix, and value matrix.

[0090] In step 630, after obtaining the query matrix, key matrix, and value matrix, attention calculation is performed based on these matrices to inject visual feature information into the diffusion processing, thereby guiding the diffusion model to predict the corresponding noise based on visual features. Specifically, firstly, matrix multiplication is performed on the query matrix and key matrix to obtain the corresponding attention score matrix. Then, exponential normalization is performed on the attention score matrix to obtain the corresponding attention weight matrix. The value matrix is ​​then weighted based on the attention weight matrix to obtain the result of the cross-attention processing. Subsequently, the result of the cross-attention processing is nonlinearly mapped through the feedforward neural network in the encoding layer to obtain the first output of the encoding layer.

[0091] In the embodiments disclosed in steps 610 to 630, cross-attention processing is performed based on the audio features to be diffused and the visual features, thereby injecting information from the visual features into the diffusion process. This allows the diffusion model to perform diffusion processing on the audio features to be diffused according to the visual features, thus determining the corresponding first output of the coding layer based on the visual features. During the cross-attention processing, attention weights can adaptively learn which parts of the visual features require focus. Moreover, even if the visual features are long, long-distance dependencies within the visual features can be captured, thereby improving the accuracy of the first output. Furthermore, compared to splicing or fully connected layers fusing multimodal information, the cross-attention approach reduces computation by sharing parameters (such as query weight matrices) while maintaining flexibility. Additionally, through attention weight allocation, noise or irrelevant information in the source sequence can be suppressed, focusing on key parts.

[0092] Furthermore, when the diffusion model is based on a diffusion converter architecture, it includes multiple diffusion converter modules. Each module comprises a multilayer perceptron, a multi-head attention mechanism, and a feedforward neural network. Intermediate data corresponding to the audio features to be diffused is input into the diffusion converter module, undergoes multi-head attention processing, and then the feedforward neural network processes the output of the multi-head attention mechanism. The diffusion converter module also includes multiple neural network layers for dimensionality scaling and offsetting of the inputs and outputs of the multi-head attention mechanism and the feedforward neural network. These neural network layers are connected to the multilayer perceptron. The diffusion model first encodes the visual features and the step numbers corresponding to the denoising steps into corresponding embedding vectors through an embedding layer. Then, in each diffusion converter module, the multilayer perceptron generates multiple dimensionality scaling and offset parameters based on these embedding vectors. These parameters are then used to scale and offset the inputs and outputs of the multi-head attention mechanism and the feedforward neural network, injecting visual feature information into the diffusion process, enabling the diffusion model to diffuse the audio features to be diffused based on the visual features.

[0093] Referring to Figure 7, in one embodiment, step 320 may further include:

[0094] Step 710: Obtain a first number, generate a first number of random seeds, and generate a first number of audio features to be diffused based on the first number of random seeds;

[0095] Step 720: Input the first number of audio features to be diffused into the diffusion model respectively, and perform diffusion processing on the first number of audio features to be diffused based on visual features to obtain the first number of candidate accompanying audio.

[0096] Step 730: Determine the accompanying audio from the first number of candidate accompanying audios.

[0097] In step 710, the first number represents the number of candidate accompanying audio files that the object wishes to generate based on the input target video. The first number can be the number input by the object through the object's terminal. Referring to Figures 2C to 2E, the object can input the first number by adjusting the batch size in the interface of the audio generation software on the object's terminal. Alternatively, the first number can be a preset number.

[0098] A random seed is a seed for generating random numbers. Referring to the explanation in step 320 above, the audio feature to be diffused is an audio feature composed entirely of noise, formed by random sampling based on a given probability distribution and the sampled values. The probability distribution is generated based on certain conditions. By using a random seed as the condition for the probability distribution to follow, different probability distributions will be generated for different random seeds. By randomly sampling the probability distributions generated based on different random seeds, different audio features to be diffused can be obtained.

[0099] For example, the audio feature to be diffused is an audio feature composed entirely of noise, obtained by random sampling based on a Gaussian distribution. Different mathematical expectations and variances will generate different Gaussian distributions. Using a random seed as the mathematical expectation or variance of the Gaussian distribution will result in different Gaussian distributions. Then, by randomly sampling from the Gaussian distributions obtained based on different random seeds, different audio features to be diffused can be obtained. For example, generating four random seeds, 1, 2, 3, and 4, will yield four Gaussian distributions, denoted as G_1, G_2, G_3, and G_4 respectively. Random sampling from G_1 yields one audio feature to be diffused; random sampling from G_2 yields another audio feature to be diffused, and so on, resulting in the first number of audio features to be diffused.

[0100] In step 720, after obtaining the first number of audio features to be diffused, diffusion processing is performed on each of the first number of audio features to be diffused based on visual features. Since the diffusion model predicts the corresponding noise based on visual features and the initial audio features to be diffused, and subtracts the predicted noise from the initial audio features to be diffused, thereby generating corresponding audio features for different audio to be diffused, the noise predicted in each denoising step of the diffusion process is also different, and the initial audio features to be diffused are also different. Thus, after the diffusion processing is completed, the generated audio will also have certain differences. In this way, by using the diffusion model to perform diffusion processing on the first number of audio features to be diffused based on visual features, the first number of candidate accompanying audio can be obtained.

[0101] It should be noted that when performing diffusion processing on each audio feature to be diffused, it is necessary to generate a corresponding first adjustment feature based on the semantic features of the target sound event, and adjust the first output generated by each coding layer of the diffusion model during the diffusion processing to obtain the corresponding second output, and generate the corresponding candidate accompanying audio based on the second output.

[0102] In addition, when performing diffusion processing on each audio feature to be diffused, the corresponding sound event timestamp feature can be obtained, and the second adjustment feature corresponding to each coding layer of the diffusion model can be generated based on the sound event timestamp feature. The second output of the coding layer can be adjusted in this way to obtain the corresponding third output, and the corresponding candidate accompanying audio can be generated based on the third output.

[0103] In step 730, after generating the first number of candidate accompanying audios, the generated first number of candidate accompanying audios can be displayed in the object's terminal interface. At the same time, in response to the object's selection operation, the candidate accompanying audios selected by the object are synthesized into the target video, so that the object can preview the result of synthesizing each candidate accompanying audio into the target video in the object's terminal interface, thereby determining the required accompanying audio from the first number of candidate accompanying audios.

[0104] In the embodiments disclosed in steps 710 to 730, a first number is obtained to generate a first number of different audio features to be diffused. Then, based on visual features, the first number of audio features to be diffused are diffused respectively to obtain a first number of candidate accompanying audio. Thus, multiple candidate accompanying audios corresponding to the target video are generated at once. In this way, the final required accompanying audio can be selected from the first number of candidate accompanying audios, thereby better aligning the accompanying audio and the target video with the target video in semantic and temporal dimensions.

[0105] In step 330, the first adjustment feature is obtained based on the semantic features of the target sound event and is a feature matrix required to inject the information contained in the semantic features of the target sound event into the diffusion processing process. It can be understood that during the diffusion processing, the diffusion model performs corresponding processing on the audio features to be diffused based on visual features through multiple coding layers, thereby capturing the correlation between visual features and the output of the diffusion model in different feature dimensions. Correspondingly, it is also necessary to inject the information of the semantic features of the target sound event into the diffusion processing process from different feature dimensions. Therefore, it is necessary to generate the first adjustment feature corresponding to each coding layer based on the semantic features of the target sound event. Specifically, the first adjustment feature can be obtained by processing the semantic features of the target sound event using a corresponding semantic feature adapter.

[0106] In one embodiment, the semantic feature adapter can replicate the parameters of the coding layer structure in the diffusion model. Then, the semantic feature adapter is trained to learn how to generate corresponding first adjustment features based on the semantic features of the target sound event, thus injecting the information corresponding to the semantic features of the target sound event into the diffusion processing. It is understood that because the initial semantic feature adapter is obtained by replicating the parameters and structure of each coding layer of the diffusion model, the structure of the semantic feature adapter is identical to the structure of the multiple coding layers of the diffusion model. The semantic feature adapter will contain multiple adapter layers corresponding to the multiple coding layers of the diffusion model. After the semantic features of the target sound event are input into the semantic feature adapter, the semantic features of the target sound event will be processed sequentially by each adapter layer, generating corresponding intermediate data. The intermediate data generated by each adapter layer is the first adjustment feature of the corresponding coding layer. Specifically, the details of how the semantic feature adapter generates the corresponding first adjustment features based on the semantic features of the target sound event, and the training process of the semantic feature adapter, will be explained later and will not be elaborated here.

[0107] The first output refers to the output obtained by each coding layer of the diffusion model after processing its corresponding input. It can be understood that in each denoising step of the diffusion process, multiple coding layers of the diffusion model need to sequentially perform corresponding processing on the audio features to be diffused. After each coding layer performs its processing task on the audio features to be diffused, it generates corresponding intermediate data, which is the first output of the coding layer. For example, referring to Figure 8, the input of the coding layer is denoted as x. After the coding layer processes the input x, it obtains the output y. This output y serves as the input to other coding layers after the first coding layer in the diffusion model; this output y is the first output of the coding layer.

[0108] It should be noted that, referring to Figure 5, in the diffusion model of the Unet architecture, the output of each coding layer serves not only as the input to the corresponding decoding layer but also as the input to the next coding layer. In this case, the first adjustment feature is only used to adjust the output from the coding layer to the corresponding decoding layer, and not to adjust the output from the coding layer to the next coding layer. For example, referring to Figure 5, in coding layer 1, after processing the audio features to be diffused based on visual features, the resulting output is simultaneously input to decoding layer 1 and coding layer 2. At this time, based on the corresponding first adjustment feature, the data output from the coding layer to decoding layer 1 is adjusted, but the output from coding layer 1 to coding layer 2 does not need to be adjusted.

[0109] Specifically, in one embodiment, the first adjustment feature and the first output of the corresponding coding layer may be superimposed to adjust the first output based on the first adjustment feature, thereby injecting the information of the semantic features of the target sound event into the diffusion process, so that the accompanying audio finally generated by the diffusion model can contain the audio segments corresponding to all sound events represented by the semantic features of the target sound event.

[0110] In the embodiments disclosed in steps 310 to 330, during the diffusion processing, semantic features of target sound events corresponding to the target video are obtained. Based on these semantic features, first adjustment features corresponding to each coding layer of the diffusion model are generated. These features are then used to adjust the first output generated by each coding layer of the diffusion model during the diffusion processing of the speech features to be diffused. The intermediate data of the diffusion processing is then fine-tuned, thereby injecting the information reflected by the semantic features of the target sound events into the diffusion processing process. This ensures that the diffusion model can capture all sound events in the target video during the diffusion processing. Consequently, the accompanying audio finally generated by the diffusion model can contain audio segments corresponding to all target sound events in the target video, ensuring that the generated accompanying audio and the target video are semantically accurately aligned.

[0111] The above provides an overall description of steps 310 to 330. Since step 320 has already been described in detail above, the following will only describe the details of steps 310 and 330 separately.

[0112] Detailed description of step 310:

[0113] In step 310, the visual features of the target video are extracted, and the semantic features of the target sound events in the target video are obtained;

[0114] In one embodiment, the sound events identified from the target video may include some sound events corresponding to noise or ambient sounds. For example, in a rainy scene, the sound event of rain will be identified, and the generated accompanying audio will contain the sound of rain. However, for the target video, the rain sound may be unnecessary, and it may even affect the sound effects corresponding to other sound events in the target video. In this case, it is necessary to filter out these sound events. Based on this, in one embodiment, referring to FIG9, step 310, obtaining the semantic features of the target sound events in the target video includes:

[0115] Step 910: Identify sound events from the target video and obtain the target frame containing the sound events;

[0116] Step 920: Input the target frame and the context frame of the target frame in the target video into the importance prediction model, and predict the importance of the sound event through the importance prediction model;

[0117] Step 930: Obtain the sound intensity corresponding to the sound event;

[0118] Step 940: Based on importance and sound intensity, determine the target sound event among the identified sound events;

[0119] Step 950: Obtain the semantic features of the target sound event.

[0120] In step 910, a sound event refers to an event in the target video that may produce a corresponding sound effect. It can include all sound events identified from the target video. After identifying these sound events, the video frame corresponding to each sound event is obtained from the target video as the target frame. For example, if the 5th second of the target video shows a rain scene, and the sound event of rain is identified based on this scene, then the video frame at the 5th second of the target video is taken as the target frame corresponding to this sound event. It should be noted that a sound event may correspond to multiple video frames; in this case, the target frame can be the first video frame where the sound event occurs.

[0121] In step 920, the importance prediction model can be a pre-trained model used to predict the importance of sound events, where importance represents the correlation between the sound event and the content of the target video. The context frame can be multiple video frames within a time interval centered on the target frame. For example, if the target frame is the video frame corresponding to the 5th second in the target video, the context frame can be multiple video frames within the time interval from one second before the target frame to one second after the target frame, i.e., multiple video frames within the time interval from the 4th to the 6th second of the target video. Exemplarily, a corresponding time interval can be determined based on the position of the target frame in the target video, and then uniform or random sampling can be performed within this time interval to obtain the corresponding context frame.

[0122] After acquiring the context frames, the target frame and context frames are input together into the importance prediction model to predict the importance of sound events in the target video. This helps identify sound events with lower importance in the target video. It's understandable that a target video often contains multiple sound events, each with varying degrees of importance. For example, if the target video shows a person playing an instrument indoors, and a few frames show rain outside through a window during the video's perspective shifts, then the target video will identify the sound events of the person playing the instrument and the rain. However, the rain sound event has a low correlation with the content of the target video. Adding the rain sound effect to the accompanying audio would become noise, affecting the sound of the person playing the instrument. In this case, the target frame and context frames of the rain sound event are input into the importance prediction model, which then predicts that this sound event has low importance.

[0123] In step 930, sound intensity can refer to the loudness corresponding to a sound event. It's understandable that in the target video, some sound events, while not of high importance, may have high sound intensity. If the generated accompanying audio doesn't include the sound effects for these sound events, it may lead to a poor viewing experience. For example, if the target video shows a person playing an instrument indoors, and during the video's perspective shifts, a few frames show a view of thunder and lightning outside through a window, then the target video will identify the sound events of the person playing the instrument and the thunder and lightning. Although the thunder and lightning sound event is not highly correlated with the content of the target video and its duration is short, if the target video plays the corresponding thunder and lightning scene without corresponding sound effects, it will still lead to a poor viewing experience for the target video viewer. Therefore, in addition to predicting the importance of each sound event in the target video, it is also necessary to determine the sound intensity of each sound event to identify the target sound events.

[0124] Specifically, various sound events can be pre-classified, and a corresponding sound intensity range can be set for each category. For example, the sound of a rigid object colliding, such as a hammer blow, often has a high sound intensity and can be classified as one category; the sound of an animal calling can also be classified as one category, and so on. After identifying the corresponding sound events from the target video, the category to which the identified sound event belongs is determined, and the sound intensity corresponding to the sound event is determined based on the sound intensity range corresponding to that category.

[0125] In step 940, specifically, a first score can be determined based on the importance of the sound event; the higher the importance of the sound event, the higher the first score. A second score can be determined based on the sound intensity of the sound event; the higher the sound intensity of the sound event, the higher the second score. The first and second scores are weighted and summed based on preset coefficients to obtain the total score for the sound event. When the total score of the sound event is greater than a given score threshold, the sound event is determined to be the target sound event. This filters out sound events with low importance and low corresponding sound intensity, i.e., some sound events that may correspond to the background noise of the target video, preventing these unimportant background noises from affecting the sound effects of other sound events in the target video, thereby improving the quality of the generated accompanying audio.

[0126] In step 950, after determining the target sound event, text describing the target sound event, i.e., the semantic description of the sound event, is generated, and the semantic description of the sound event is encoded through a semantic feature generation model to obtain the corresponding semantic features of the target sound event.

[0127] In steps 910 to 950, after identifying sound events in the target video, the importance of each sound event is predicted, and the corresponding sound intensity is determined. Importance represents the correlation between the sound event and the content of the target video, and sound intensity can refer to the loudness of the sound event, in order to improve the viewing experience of the video. Based on the importance and sound intensity of the sound events, the target sound event is determined from all the identified sound events, thereby filtering out sound events with low importance and low sound intensity in the target video. Then, semantic features of the target sound event are generated based on the target sound event. This allows for the reduction of unnecessary noise in the accompanying audio generated subsequently based on the semantic features of the target sound event, improving the quality of the generated accompanying audio.

[0128] In one embodiment, referring to FIG10, step 310 may further include:

[0129] Step 1010: Decompose the target video into video frames;

[0130] Step 1020: Input the decomposed video frames and preset prompts into the visual language model so that the visual language model can generate a semantic description of the sound event. The preset prompts are used to prompt the visual language model to generate a semantic description of the sound event based on the video frames.

[0131] Step 1030: Encode the semantic description of the sound event using a semantic feature generation model to obtain the semantic features of the target sound event.

[0132] In step 1010, a video frame refers to a single frame image in the target video. Specifically, all video frames of the target video can be extracted directly; or, the target video can be decomposed into multiple segments, and samples can be taken from each segment to obtain multiple video frames corresponding to the target video; or, the target video can be uniformly sampled based on a given time interval to extract the corresponding video frames from the target video; in addition, the difference between two adjacent video frames in the target video can be calculated to determine whether there is a large change between two adjacent video frames, thereby extracting relatively key video frames in the target video, thus decomposing the target video into video frames.

[0133] In step 1020, the visual language model is a multimodal model capable of capturing and processing information from both image and text modalities to generate corresponding text output. For example, the visual language model could be a multimodal large language model supporting input images, or a multimodal model specifically designed for visual language tasks, such as the Mini-Gemini model (used to enhance the capabilities of multimodal visual language models). The visual language model can accurately capture the relationships between multiple input image information and thereby identify events occurring in the multiple input images. The preset prompt is used to guide the visual language model to detect sound events contained in the input video frames. For example, the preset prompt could be "Please identify the sound events contained in this set of images." Referring to Figure 4, the extracted video frames and the preset prompt are input into the visual language model, enabling the model to identify sound events within the input video frames and generate corresponding semantic descriptions of the sound events. Thus, by utilizing the visual language model's ability to process image information, sound events contained in the target video are identified from the video frames. It can be understood that the semantic description of the sound events is a text used to describe the sound events contained in the target video.

[0134] In step 1030, the semantic feature generation model is a text modality encoder that encodes text information. It should be noted that the semantic feature generation model can encode text information into text features, which have a high similarity to video features of videos with the same semantic information. For example, if the text is "A puppy is barking," and the corresponding video is also "A puppy is barking," then the text features obtained by encoding this text through the semantic feature generation model are highly similar to the video features obtained by encoding the video "A puppy is barking." After obtaining the semantic description of the sound event, referring to Figure 4, the semantic feature generation model encodes the semantic description of the sound event to obtain the corresponding target sound event semantic features. Specifically, the training process of the semantic feature generation model will be explained later and will not be elaborated here.

[0135] In the embodiments disclosed in steps 1010 to 1030, by decomposing the target video into multiple video frames, and then inputting the video frames and preset prompts into a visual language model, the image processing capability of the visual language model can be used to accurately capture the correlation between the multiple input video frames, and thereby identify the sound events occurring in the multiple input video frames. This enables the identification of sound events in the target video based on video frames, generating corresponding semantic descriptions of sound events, and thereby generating corresponding semantic features of the target sound events, thus improving the accuracy of the semantic features of the target sound events.

[0136] In one embodiment, referring to FIG11, after step 1030, the method further includes:

[0137] Step 1110: Obtain the cue word set and encode the cue word set into corresponding cue word features; wherein, the cue word set includes positive cue words and negative cue words, positive cue words are used to prompt the diffusion model to correspond to the first reference sound event in the generated accompanying audio, and negative cue words are used to prompt the diffusion model to correspond to the second reference sound event in the generated accompanying audio.

[0138] Step 1120: Concatenate the prompt word features to the target sound event semantic features to update the target sound event semantic features.

[0139] In step 1110, the positive cue word is used to indicate the first reference sound event that the diffusion model needs to add to the accompanying audio. The first reference sound event is the sound event that the object wants to have a corresponding audio segment in the generated accompanying audio. The negative cue word is used to indicate the second reference sound event that the diffusion model cannot correspond to in the generated accompanying audio. The second reference sound event is the sound event that does not have a corresponding sound effect in the generated accompanying audio. For example, if the target video is a video of a pet dog playing, then the positive cue word could be: dog barking. The negative cue word can be a pre-set cue word used to indicate sound events corresponding to sounds that are mostly unrelated to the target video. For example, the negative cue word could be: noise, background music, etc. In addition, the negative cue word can also be used to indicate sounds that may be related to the target video, but that the object does not want to appear in the accompanying audio in most scenarios, such as: sharp and piercing sounds, etc.

[0140] Positive and negative prompts can also be input by the object through its terminal. For example, referring to Figure 12, the object can select the option to input prompts in the interface of the audio generation software. Then, text input boxes for inputting positive and negative prompts will be displayed in the interface of the audio generation software, and the object can input the corresponding positive and negative prompts in the corresponding text input boxes.

[0141] After obtaining the positive and negative cue words, which are also text modal information, similar to the semantic description of sound events, a semantic feature generation model can be used to encode the positive and negative cue words into corresponding cue word features. It should be noted that since the negative cue words are used to indicate a second reference sound event that cannot be corresponding to the accompanying audio, when encoding the negative cue words into cue word features, the negative cue words and their corresponding negative cue templates need to be encoded together. For example, the negative cue template could be "Does not need to include the following sound events:". This avoids misidentifying the negative cue words as positive cue words when injecting the cue word feature information into the diffusion process, which would result in the final generated accompanying audio carrying audio segments of the second reference sound event corresponding to the negative cue words.

[0142] After obtaining the cue word features, since both the cue word features and the target sound event semantic features are used to describe the sound events corresponding to the accompanying audio, the cue word features can be concatenated with the target sound event semantic features to update the target sound event semantic features. Subsequently, based on the updated target sound event semantic features, a corresponding first adjustment feature is generated. This injects the information carried by the positive and negative cue words into the diffusion processing, ensuring that the accompanying audio generated by the diffusion model can carry audio segments of the sound events corresponding to the positive cue words, but not audio segments of the sound events corresponding to the negative cue words.

[0143] In steps 1110 to 1120, positive and negative cue words are acquired and corresponding cue word features are generated. These cue word features are then concatenated with the semantic features of the target sound event. This allows the user to manually input positive and negative cue words, selectively injecting semantic information of some sound events into the diffusion process. Specifically, the information carried by the positive and negative cue words is injected into the diffusion process, ensuring that the accompanying audio generated by the diffusion model carries audio segments of the sound events corresponding to the positive cue words but not those corresponding to the negative cue words. This also avoids situations where the visual language model fails to accurately identify all sound events in the target video or incorrectly identifies sound events not present in the target video.

[0144] Detailed description of step 330:

[0145] In step 330, a first adjustment feature corresponding to the coding layer is generated based on the semantic features of the target sound event, and the first output of the coding layer is adjusted based on the first adjustment feature to obtain a second output, and the accompanying audio of the target video is generated based on the second output.

[0146] Referring to Figure 13, in step 330, the first adjustment feature corresponding to the encoding layer is generated based on the semantic features of the target sound event, including:

[0147] Step 1310: Obtain the initial input of the first coding layer of the diffusion model;

[0148] Step 1320: Perform zero-convolution processing on the semantic features of the target sound event, and superimpose the semantic features of the target sound event after zero-convolution processing with the initial input to obtain the superimposed semantic features of the target sound event.

[0149] Step 1330: Input the superimposed semantic features of the target sound event into the corresponding semantic feature adapter, perform zero convolution on the output of the semantic feature adapter to obtain the first adjusted feature, wherein the semantic feature adapter is trained by freezing the parameters of the diffusion model.

[0150] In step 1310, since the semantic feature adapter needs to determine the first adjustment features corresponding to each coding layer of the diffusion model based on the corresponding input, in order to adjust the first output of each coding layer, when determining the first adjustment features, it is necessary to consider not only the semantic features of the target sound event input, but also the inputs of multiple coding layers of the diffusion model. Based on this, the initial input of the first coding layer among the multiple coding layers of the diffusion model is obtained, i.e., the audio features to be diffused.

[0151] In step 1320, referring to Figure 14, a zero-convolutional layer is set at the input of the semantic feature adapter. This zero-convolutional layer is a 1*1 convolutional kernel, and its weights and biases are determined during the training of the semantic feature adapter. Before the target sound event semantic features or sound event timestamp features are input to the corresponding semantic feature adapter, they are first processed by the corresponding zero-convolutional layer. Then, they are superimposed with the initial input of the first encoding layer obtained from the diffusion model through an adder to obtain the superimposed target sound event semantic features, which are used as the input of the semantic feature adapter.

[0152] In step 1330, the semantic features of the superimposed target sound event are input into the corresponding semantic feature adapter. The semantic feature adapter is obtained by copying the structure and parameters of the encoding layer of the diffusion model, then freezing the parameters of the diffusion model for training, and optimizing the parameters of the semantic feature adapter. The semantic feature adapter has the same network structure as the encoding layer of the diffusion model, and includes adapter layers corresponding to each encoding layer of the diffusion model. After the semantic features of the superimposed target sound event are input into the semantic feature adapter, each neural network layer in the semantic feature adapter sequentially performs corresponding processing on the semantic features of the superimposed target sound event, thus obtaining the output corresponding to each adapter layer. Specifically, since the semantic feature adapter has the same neural network structure as the encoding layer of the diffusion model, the difference lies in the parameters of each layer. Therefore, the processing process performed by the semantic feature adapter on the semantic features of the superimposed target sound event is the same as the processing process performed by each encoding layer of the diffusion model on the audio features to be diffused. The difference is that the parameters in the processing are the parameters of the corresponding neural network layers. Based on this, the processing procedure performed by the semantic feature adapter on the semantic features of the superimposed target sound event can refer to the processing procedure performed by each encoding layer of the diffusion model on the audio features to be diffused when performing diffusion processing on the audio features to be diffused based on visual features in step 320 above. For example, when the diffusion model is a model based on the Unet architecture, each adapter layer of the semantic feature adapter also includes a cross-attention mechanism and other neural network layers. After the semantic features of the superimposed target sound event are input into the semantic feature adapter, the corresponding processing is performed through the neural network layers in the adapter layer, and in the cross-attention mechanism, cross-attention processing is performed based on visual features and the semantic features of the superimposed target sound event.

[0153] After obtaining the outputs of each adapter layer, since each adapter layer of the semantic feature adapter is connected to the output of the corresponding encoding layer in the diffusion model through a zero-convolutional layer, it is necessary to perform zero-convolution processing on the outputs of each adapter layer to obtain the first adjusted features corresponding to each encoding layer. Similar to step 1320, the weights and biases of these zero-convolutional layers are also determined during the training of the semantic feature adapter, which will not be elaborated further. It should be noted that since the zero-convolution processing in steps 1320 and 1330 is performed through different zero-convolutional layers, the parameters of the different zero-convolutional layers are different. That is, the weight and bias parameters when performing zero-convolution processing in steps 1320 and 1330 are different.

[0154] For example, steps 1310 to 1330 can be represented as: y c =Z(F(x+Z(c;Θ) z1 );Θc );Θ z2 );

[0155] Where x represents the initial input, i.e., the audio features to be diffused; c represents the semantic features of the target sound event; y c Indicates the first adjustment feature; Θ c F(Θ) represents the parameters of each adapter layer of the semantic feature adapter. c ) indicates that each adapter layer is based on the corresponding parameter Θ c Perform corresponding processing on the input; Θ z1 Z(c; Θ) represents the parameters of the zero convolutional layer at the input of the semantic feature adapter. z1 ) indicates based on parameter Θ z1 Perform zero-convolution processing on the semantic features of the input target sound event; Θ z2 Z(Θ) represents the parameters of the zero convolutional layer connected to the output of the adapter layer of the semantic feature adapter. z2 ) indicates based on parameter Θ z2 The output of the adapter layer is subjected to zero convolution.

[0156] In the embodiments disclosed in steps 1310 to 1330, the semantic features of the input target sound event are subjected to zero-convolution processing, and the zero-convolution-processed semantic features of the target sound event are superimposed with the initial inputs of multiple coding layers of the diffusion model to obtain superimposed semantic features of the target sound event. Then, the superimposed semantic features of the target sound event are input into a trained semantic feature adapter, thereby performing corresponding processing on the superimposed semantic features of the target sound event through each adapter layer to obtain the output of each adapter layer. The outputs of each adapter layer are then subjected to zero-convolution processing. Thus, the semantic feature adapter generates first adjustment features corresponding to each coding layer based on the semantic features of the target sound event. Zero-convolution processing does not destroy the initial structure of the semantic features of the target sound event, ensuring that the diffusion model can learn the semantics of each sound event. Furthermore, zero-convolution processing can fuse the semantic features of the target sound event with the outputs of the coding layers, avoiding feature conflicts caused by direct superposition, and enabling semantic information to simultaneously influence feature generation at different levels. This improves the semantic consistency of information across different modalities while enhancing detail richness, thereby improving the accuracy of the first adjustment features.

[0157] Steps 1310 to 1330 above illustrate the process of generating the first adjusted feature based on the semantic features of the target sound event. The subsequent process of generating the second adjusted feature based on the sound event timestamp features can refer to the detailed process of steps 1310 to 1330. That is, the sound event timestamp features after zero convolution are superimposed with the initial input and input into the timestamp feature adapter for corresponding processing. Then, the outputs of each adapter layer are zero-convolutioned through the corresponding zero convolution layers to obtain the second adjusted features corresponding to each coding layer. This will not be elaborated further here.

[0158] In one embodiment, referring to FIG15, the accompanying audio generation method further includes:

[0159] Step 1510: Obtain the first weight coefficient corresponding to the first adjusted feature;

[0160] Step 1520: The first adjusted feature is weighted based on the first weight coefficient to obtain the weighted first adjusted feature, and the weighted first adjusted feature is superimposed with the first output of the corresponding coding layer.

[0161] In step 1510, the weighting coefficient is used to adjust the diffusion processing based on the semantic features of the target sound event. In one embodiment, the weighting coefficient can be input by the object through the object's terminal. For example, referring to FIG16, the interface of the audio generation software is provided with a slider for controlling the weighting coefficient. The object adjusts the slider in the interface of the audio generation software to form the input weighting coefficient. The first weighting coefficient is the weighting coefficient that adjusts the first adjustment feature.

[0162] In step 1520, after obtaining the corresponding first weight coefficient, the first adjustment feature is weighted based on the first weight coefficient. Specifically, the first adjustment feature is a matrix with the same shape as the output of the coding layer, while the first weight coefficient is an input constant. By multiplying each element of the first weight coefficient by the first adjustment feature, the first adjustment feature is weighted based on the first weight coefficient to obtain the weighted first adjustment feature. Then, the weighted first adjustment feature is superimposed on the first output of the corresponding coding layer, thereby adjusting the first output of the coding layer based on the weighted first adjustment feature. Specifically, steps 1510 to 1520 can be expressed as: y = y' + λy c Where y' represents the first output of the coding layer, λ represents the first weight coefficient, and y c Let y represent the first adjustment feature, and y represent the second output based on the coding layer.

[0163] In the embodiments disclosed in steps 1510 to 1520, by obtaining the first weight coefficient corresponding to the first adjustment feature, the first adjustment feature is weighted, and then the weighted first adjustment feature is superimposed on the first output of the corresponding coding layer. In this way, the object can control the adjustment range when adjusting the diffusion process based on the semantic features of the target sound event by adjusting the size of the weight coefficient, so as to better control the process of accompanying audio generation according to the needs of the object.

[0164] Implementation method of adjusting diffusion processing based on sound event timestamp features:

[0165] In one embodiment, the target video may contain short sound events, such as the sound of two rigid objects colliding, like a hammer striking. When generating accompanying audio, if the sound effects corresponding to these short sound events are not accurately aligned with the corresponding video frames in terms of timestamps, it will lead to significant audio-visual desynchronization. For example, if the target video contains a hammering scene, but the generated accompanying audio's hammering sound effect occurs 0.5 seconds before the hammering scene, the quality of the accompanying audio will be relatively low. Therefore, in one embodiment, referring to Figure 17, the accompanying audio generation method further includes:

[0166] Step 1710: Obtain the audio event timestamp features of the target video;

[0167] Step 1720: Generate a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature; adjust the second output based on the second adjustment feature to obtain the third output;

[0168] Step 1730: Generate accompanying audio for the target video based on the third output, so that the accompanying audio includes audio segments corresponding to the semantic features of the target sound event according to the timestamps corresponding to the sound event timestamp features.

[0169] In step 1710, the sound event timestamp feature is a feature used to characterize the time node where sound events occur in the target video. Based on the sound event timestamp feature, the time node at the beginning of each sound event in the target video can be determined. For example, if a hammering sound event occurs at the 5th second of the target video, then the 5th second of the target video is the timestamp corresponding to this hammering sound event. After identifying the timestamp of each sound event in the target video, these timestamps are encoded to obtain the corresponding sound event timestamp feature. The details of step 1710 will be explained later and will not be elaborated here.

[0170] In step 1720, after obtaining the sound event timestamp features, these features are input into a trained timestamp feature adapter. The adapter processes the timestamp features to determine the second adjustment features corresponding to each coding layer. The timestamp feature adapter is trained by freezing the parameters of the diffusion model. The trained adapter learns how to accurately process the input sound event timestamp features into second adjustment features that inject the information contained within them into the diffusion process. Based on this, after obtaining the second adjustment features, the output of the coding layers of the diffusion model is adjusted, ensuring that the accompanying audio generated by the diffusion model is accurately aligned with the target video in the time dimension. The training process of the timestamp feature adapter will be explained later and will not be elaborated here.

[0171] Specifically, the structure of the timestamp feature adapter is consistent with that of the semantic feature adapter. The details of generating the second adjustment feature corresponding to the coding layer based on the timestamp feature of the sound event through the timestamp feature adapter can be found in steps 1310 to 1330 above, which describes the process of determining the first adjustment feature corresponding to each coding layer based on the semantic feature of the target sound event. These details will not be elaborated here.

[0172] After obtaining the second adjustment features corresponding to each coding layer, the second adjustment features are superimposed on the second output corresponding to the coding layer to inject the timestamp information contained in the sound event timestamp features into the diffusion process. This allows the accompanying audio generated by the diffusion model to be accurately aligned with the target video in the time dimension. For example, in the diffusion model of the Unet architecture, each coding layer passes its corresponding first output to the corresponding decoding layer for decoding, thereby predicting the noise corresponding to the current denoising step. After adjusting the first output of the coding layer to the second output using the first adjustment features, the second adjustment features can be superimposed on the second output passed from the coding layer to the decoding layer to obtain the third output, which is then used as the input to the corresponding decoding layer of the coding layer.

[0173] In step 1730, after adjusting the second output of the encoding layer to obtain the third output, the neural network layer in the diffusion model that receives the output of the encoding layer performs subsequent processing based on the third output to predict the corresponding noise and gradually subtract the noise predicted in each denoising step from the audio features to be diffused, generating accompanying audio. For example, in the diffusion model of the Unet architecture, the third output serves as the input to the decoding layer corresponding to the encoding layer. The decoding layer decodes based on the input third output to predict the noise of the corresponding denoising step and uses this to denoise the audio to be diffused, generating accompanying audio.

[0174] In the embodiments disclosed in steps 1710 to 1730, by obtaining the sound event timestamp features corresponding to the target video, and then determining the second adjustment features corresponding to each coding layer based on the sound event timestamp features, and adjusting the second output corresponding to each coding layer based on the second adjustment features to obtain the third output, the intermediate data of the diffusion model in the diffusion processing process is adjusted, and the timestamp information contained in the sound event timestamp features is injected into the diffusion processing process of the diffusion model, so that the diffusion model performs subsequent processing based on the third output. Thus, the final generated accompanying audio can be more accurately aligned with the target video in the time dimension.

[0175] For example, referring to Figure 18, the left side of Figure 18 shows the audio generation result when the diffusion process is not adjusted by the sound event timestamp feature, and the right side shows the audio generation result when the diffusion process is adjusted by the sound event timestamp feature. It can be seen that in scenarios such as shooting, hammering, and sharpening knives, after adjusting the diffusion process by the sound event timestamp feature, the time nodes of the sound events and the corresponding video frames can be better aligned.

[0176] In one embodiment, referring to FIG19, step 1710 includes:

[0177] Step 1910: Identify target audio events from the target video;

[0178] Step 1920: Obtain the duration of the target sound event;

[0179] Step 1930: Obtain the event type of the target sound event;

[0180] Step 1940: If the duration and event type meet the predetermined conditions, obtain the audio event timestamp features of the target video.

[0181] In step 1910, an event detection model can be used to detect events in the target video, and then the detected events can be classified to identify the events that will produce corresponding sounds as target sound events.

[0182] In step 1920, the duration of the target sound event refers to the duration of the target sound event within the target video. Specifically, the duration of the target sound event can be determined by detecting the timestamps corresponding to the start and end frames of the target sound event within the target video. For example, if the timestamp of the start frame of a sound event in the target video is 5 seconds and the timestamp of the end frame is 5.5 seconds, then the duration of the sound event is 0.5 seconds.

[0183] In step 1930, the event type is used to characterize the classification result to which the target sound event belongs. Specifically, a classification table can be pre-established, which can be pre-constructed based on expert experience. The event type of the detected sound event is determined by comparing with the classification table. The event type can include: a first event type, a second event type, and a third event type. Specifically, the first event type refers to the sound event type that requires accurate alignment. The first event type can be a sound event with a short duration and a relatively prominent corresponding sound effect (e.g., a high pitch or loudness), such as a lightning strike. The second event type refers to the sound event type that requires consideration of the duration to determine whether alignment is needed. The second event type of sound event does not require accurate alignment if the duration is long, but requires accurate alignment if the duration is short. For example, the sound event of a hammering sound requires accurate alignment if it is a single or a few hammerings, but does not require accurate alignment for long-term continuous hammering. The third event type is the sound event type that does not require accurate alignment. It refers to sound events that are often long in duration and have a less prominent sound effect, such as rain.

[0184] In step 1940, the predetermined condition is used to determine whether the sound effect corresponding to the target sound event needs to be accurately aligned with the target video. When the target video contains a target sound event corresponding to the first event type, it is determined that the predetermined condition is met, and the sound event timestamp feature of the target video is obtained. When the target video contains a target sound event corresponding to the second sound event type, and the duration of the target sound event is less than a given duration threshold, it is determined that the predetermined condition is met, and the sound event timestamp feature of the target video is obtained. The steps for obtaining the sound event timestamp feature of the target video will be explained later and will not be elaborated here.

[0185] In the embodiments disclosed in steps 1910 to 1940, target sound events in the target video are identified, and then the duration and event type of the target sound events are detected. Subsequently, it is determined whether the target video contains sound events that require accurate alignment based on predetermined conditions. When the duration and event type meet the predetermined conditions, it is determined that the timestamp features of the sound events corresponding to the target video need to be obtained. This is used to generate corresponding second adjustment features based on the sound event timestamp features in subsequent steps, thereby injecting the information contained in the sound event timestamp features into the diffusion processing process, ensuring that the generated accompanying audio can be accurately aligned with the target video in the time dimension.

[0186] In one embodiment, the target video may contain objects, which can be anthropomorphic animals or people. In the target video, the objects may converse with each other, or the objects may make corresponding sounds through their mouths, such as laughter or shouts. These sounds and the objects' mouth movements need to be accurately aligned in time. Therefore, it is necessary to obtain the corresponding sound event timestamp features. Based on this, in one embodiment, referring to FIG20, the accompanying audio generation method further includes:

[0187] Step 2010: If the target sound event is an object speech event, identify the mouth size of the object in the frame corresponding to the object speech event in the target video;

[0188] Step 2020: If the size of the object's mouth exceeds the predetermined size, obtain the audio event timestamp features of the target video.

[0189] In step 2010, an object speech event refers to a sound event in which an object makes a sound through its mouth, such as laughter, crying, or even conversation. For example, if the target video contains a scene of a child laughing, this scene corresponds to an object speech event. When an object speech event is detected in the target video, the size of the corresponding object's mouth is detected to determine whether the object's mouth can be displayed relatively clearly in the target video.

[0190] In step 2020, it's understood that if the object's mouth size is no larger than a predetermined size—meaning the object's mouth is relatively inconspicuous in the target video, or even if the object is facing away from the video—there won't be significant audio-visual asynchrony even without aligning the corresponding sound from the mouth with the video frame using timestamp features. However, when the mouth size is larger than the predetermined size, it indicates that the object's mouth is clearly visible in the target video frame. In this case, it's necessary to precisely align the corresponding sound effects and mouth movements in the time dimension. Therefore, when the object's mouth size exceeds the predetermined size, the timestamp features of the sound events in the target video are obtained to ensure that the generated accompanying audio is accurately aligned with the target video in the time dimension, avoiding significant audio-visual asynchrony and improving the quality of the generated accompanying audio.

[0191] In steps 2010 to 2020, the presence of an object speech event in the target video is detected. When an object speech event exists, the size of the object's mouth is detected. If the size of the object's mouth is larger than a predetermined size, the sound event timestamp feature of the target video is obtained to avoid significant audio-visual asynchrony. This is then used to generate a corresponding second adjustment feature based on the sound event timestamp feature and adjust the second output of the coding layer. This injects the information of the sound event timestamp feature into the diffusion process, ensuring that the generated accompanying audio is also accurately aligned with the target video in the time dimension, thereby improving the quality of the generated accompanying audio.

[0192] In one embodiment, referring to FIG21, step 1710 includes:

[0193] Step 2110: In response to the first operation of the target terminal on the target video, determine a plurality of first annotation information, wherein the first annotation information is used to annotate the time nodes in which sound events occur in the target video;

[0194] Step 2120: Determine the sound event timestamp features based on multiple first annotation information.

[0195] In step 2110, the first operation is the operation of the object terminal to annotate the target video, and the first annotation information is used to characterize the time node when the object is annotated in the target video by the object terminal.

[0196] Specifically, referring to Figure 22, in the interface of the audio generation software, the object can select the option to annotate a timestamp. Then, the target video will be played in the audio generation software interface, displaying the current playback frame at the target video's time point. For example, in Figure 22, the current playback frame is at the 5th second of the target video. The audio generation software interface also includes an option for annotating timestamps, i.e., adding a timestamp. When the object clicks this option in the audio generation software interface, a corresponding first annotation information will be generated, marking the time point of the currently playing frame of the target video. For example, when the target video is played to the 5th second, if the object clicks the "add timestamp" option in the audio generation software interface, a first annotation information corresponding to the 5th second of the target video will be generated.

[0197] In step 2120, each first annotation information is actually a timestamp information. Multiple first annotation information are concatenated, and then the concatenated first annotation information is encoded to obtain the corresponding sound event timestamp feature.

[0198] In the embodiments disclosed in steps 2110 to 2120, by obtaining the first annotation information of the object input, the time nodes of multiple sound events in the target video are determined, and corresponding sound event timestamp features are generated accordingly. This allows for selective annotation of the time nodes where sound events occur in the target video, preventing the model from failing to recognize the timestamps corresponding to all sound events in the target video. Subsequently, sound event timestamp features are generated based on the first annotation information. These features are then processed by a corresponding feature adapter to generate corresponding second adjustment features. These features are then used to adjust the second outputs corresponding to each encoding layer of the diffusion model during the diffusion process. This injects the timestamp information corresponding to each sound event in the target video into the diffusion process, enabling the accompanying audio generated by the diffusion model to be more accurately aligned with the target video in the time dimension.

[0199] In one embodiment, referring to FIG23, obtaining the audio event timestamp features of the target video further includes:

[0200] Step 2310: Detect sound events based on the visual features of the target video to determine at least one sound event in the target video, and determine at least one time node corresponding to the at least one sound event in the target video.

[0201] Step 2320: Determine the timestamp features of the sound event based on at least one time node.

[0202] In step 2310, the time node corresponding to the sound event in the target video refers to the time node in the target video corresponding to the moment the sound event occurs. For example, if there is a scene of a dog barking at the 5th second of the target video, then the 5th second of the target video is the time node corresponding to the sound event of the dog barking. By detecting the time nodes of various sound events in the target video, some short sound events in the target video can be accurately captured, such as the time nodes corresponding to hammering sounds. In this way, after generating sound event timestamp features based on the detected time nodes and adjusting the second output according to the sound event timestamp features, the sound effects corresponding to some short sound events in the generated accompanying audio can be accurately aligned with the scene corresponding to the sound event in the time dimension, avoiding the problem of audio-visual desynchronization.

[0203] Specifically, referring to Figure 24, the object can choose to automatically obtain timestamps in the interface of the audio generation software. Then, by calling a pre-trained multimodal model for detecting sound events based on visual information, sound events are detected on the visual features to determine the time nodes of each sound event in the target video. For example, event detection is performed on the target video to identify the events occurring in it. The detected events are then classified to determine whether they are sound events. For instance, when a hammering event is detected, since hammering events are often accompanied by hammering sounds, this event can be identified as a sound event. Simultaneously, the time when this event occurs in the target video is determined, thus identifying the time node corresponding to the sound event.

[0204] In another embodiment, the time points of sound events in the target video can also be automatically detected by a timestamp detector, for example, by using the librosa.onset.onset_detect module (used to detect the starting point of notes) to detect the time points of sound events in the sample video.

[0205] In step 2320, by performing sound event detection on the target video and determining at least one time node corresponding to at least one sound event contained in the target video, the information of these time nodes is encoded to obtain the corresponding sound event timestamp features.

[0206] In the embodiments disclosed in steps 2310 to 2320, sound event detection is performed on the target video to determine the start time of each sound event, which is then used as the corresponding time node. This time node information is then encoded into sound event timestamp features. These sound event timestamp features accurately characterize the time nodes of each sound event contained in the target video. Subsequently, a second adjustment feature is generated based on the sound event timestamp features, and this feature is used to adjust the second output. This injects the information of the sound event timestamp features into the diffusion processing process, enabling the generated accompanying audio to be more accurately aligned with the target video frame in the time dimension.

[0207] In one embodiment, referring to FIG25, step 1730 includes:

[0208] Step 2510: Obtain the first weight coefficient corresponding to the first adjustment feature and the second weight coefficient corresponding to the second adjustment feature, wherein the sum of the first weight coefficient and the second weight coefficient is not greater than the first threshold.

[0209] Step 2520: The first adjusted feature and the second adjusted feature are weighted based on the first weight coefficient and the second weight coefficient respectively to obtain the weighted first adjusted feature and the weighted second adjusted feature. The weighted first adjusted feature and the weighted second adjusted feature are accumulated to the output of the corresponding coding layer.

[0210] In step 2510, it is understood that when it is necessary to adjust the first output of the coding layer simultaneously based on the semantic features of the target sound event and the timestamp features of the sound event, since the semantic feature adapter used to generate the first adjustment feature and the timestamp feature adapter used to generate the second adjustment feature are trained separately, the process of generating the first adjustment feature based on the semantic features of the target sound event and the process of generating the second adjustment feature based on the timestamp features of the sound event are independent of each other. Since both the first and second adjustment features are feature matrices with the same shape as the first output, simultaneously superimposing the first and second adjustment features onto the first output of the corresponding coding layer may cause distortion of the first output due to an excessively large sum of adjustment amplitudes of the first and second adjustment features, resulting in a decrease in the quality of the final generated accompanying audio. Therefore, in this embodiment, a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature are obtained. The second weight coefficient is a weight coefficient used to adjust the second adjustment feature. The sum of the first and second weight coefficients is controlled to be no greater than a preset first threshold, which can be 1, thereby controlling the sum of the adjustment amplitudes of the first and second adjustment features.

[0211] In step 2520, after determining the first weighting coefficient and the second weighting coefficient, the first adjustment feature is weighted based on the first weighting coefficient to obtain the weighted first adjustment feature. The second adjustment feature is then weighted based on the second weighting coefficient to obtain the weighted second adjustment feature. Finally, the weighted first and second adjustment features are superimposed on the first output of the corresponding coding layer. This avoids distortion of the first output due to an excessively large sum of adjustment amplitudes when simultaneously adjusting the first output using both the first and second adjustment features. Furthermore, when injecting both the semantic features and timestamp features of the target sound event into the diffusion processing, the injected information from these features can be avoided from affecting the audio generation capability of the diffusion model.

[0212] In the embodiments disclosed in steps 2510 to 2520, a first weighting coefficient and a second weighting coefficient are obtained, and the first adjustment and the second adjustment are weighted based on the first weighting coefficient and the second weighting coefficient respectively to obtain a weighted first adjustment feature and a weighted second adjustment feature. Then, the weighted first adjustment feature and the weighted second adjustment feature are superimposed on the first output of the corresponding coding layer to obtain a third output. The sum of the first weighting coefficient and the second weighting coefficient is not greater than a first threshold. This avoids the problem of decreased quality of the generated accompanying audio due to an excessively large sum of adjustment amplitudes when adjusting the diffusion process based on the first adjustment feature and the second adjustment feature. Therefore, the diffusion process can accurately generate accompanying audio that is accurately aligned with the target video in both time and semantic dimensions based on the semantic features and timestamp features of the target sound event.

[0213] Training process of semantic feature generation model:

[0214] Since the diffusion model is designed to generate audio based on visual features, it learns how to understand the visual features of a video and generate audio accordingly, but not how to generate audio based on text features. When encoding the semantic description of a sound event into corresponding target sound event semantic features, it is necessary to ensure a high similarity between the target sound event semantic features and the corresponding video features. This allows for more accurate injection of the target sound event semantic feature information into the diffusion process when the first output of the encoding layer is adjusted based on the first adjustment feature, enabling the diffusion model to capture the information contained within the target sound event semantic features. Therefore, the semantic feature generation model needs to be pre-trained so that it can encode a given sound event semantic description into text features with high similarity to the video features of the corresponding video. Specifically, referring to Figure 26, in one embodiment, the semantic feature generation model can be pre-trained in the following manner:

[0215] Step 2610: Obtain the first training sample set, wherein the first training sample set includes multiple video-text pairs, and each video-text pair includes corresponding sample video and sample sound event semantic descriptions.

[0216] Step 2620: Encode each sample video using the video feature generation model, and encode the semantic description of each sample sound event using the current semantic feature generation model to obtain multiple sample video features and sample sound event semantic description features.

[0217] Step 2630: Predict the first matching probability between each sample video feature and each sample sound event semantic description feature, and determine the second matching probability between each sample video feature and each sample sound event semantic description feature based on the correspondence between the sample video and the sample sound event semantic description.

[0218] Step 2640: Determine the first loss value based on the first matching probability and the second matching probability, optimize the current semantic feature generation model based on the first loss value, until the first loss value is less than the first loss threshold, and obtain the semantic feature generation model.

[0219] In step 2610, the first training sample set is a dataset constructed from sample videos and sample sound event semantic descriptions. The sample videos and sample sound event semantic descriptions in the first training sample set are in one-to-one correspondence. The sample sound event semantic description is text used to describe the video content in the corresponding sample video. For example, if the content of a sample video is a person hammering an object, the corresponding sample sound event semantic description is "hammering an object with a hammer".

[0220] In step 2620, the video feature generation model is a video modality encoder that extracts multiple sample video frames from a sample video. These sample video frames are then convolved to obtain the corresponding sample video frame features. Finally, a fully connected layer fuses these features to obtain the final sample video features. It should be noted that the video feature generation model in step 2320 is different from the visual encoder mentioned in step 310 for extracting visual features from the target video; both are pre-trained using different methods.

[0221] The current semantic feature generation model is a speech feature generation model that has not yet been fully trained. The semantic feature generation model is a text modality encoder that converts the semantic description of sample sound events into multiple lexical units, encodes each lexical unit into a corresponding feature vector, and then concatenates the feature vectors corresponding to each lexical unit to obtain the corresponding semantic description features of sample sound events. The semantic description features of sample sound events can be used to characterize the semantic features of the semantic description of sample sound events.

[0222] In step 2630, the first matching probability is a probability distribution obtained by predicting the matching probability between the semantic description features of each sample sound event and the sample video features. For example, the first training sample set includes 100 one-to-one corresponding sample videos and sample sound event semantic descriptions. After processing in step 2620, 100 sample video features and sample sound event semantic description features are obtained. Then, for each sample sound event semantic description feature, the matching probability between the sample sound event semantic description feature and the 100 sample video features is predicted; correspondingly, for each sample video feature, the matching probability between the sample video feature and the 100 sample sound event semantic description features is predicted. Specifically, the matching probability distribution between sample video features and sample audio event semantic description features can be obtained by calculating the cosine similarity score. Then, the similarity score between a single sample audio event semantic description feature and each sample video feature is exponentially normalized using the softmax function. For example, if the first training sample set includes 100 one-to-one corresponding sample videos and sample audio event semantic descriptions, for each sample audio event semantic description feature, the cosine similarity between that feature and the 100 sample video features is calculated, resulting in 100 corresponding cosine similarity scores. These 100 cosine similarity scores are then exponentially normalized, thus yielding the matching probability distribution between the sample audio event semantic description feature and each sample video feature. Similarly, the matching probability distribution between sample video features and each sample audio event semantic description feature can also be obtained. Therefore, the first matching probability can be determined.

[0223] The second matching probability is a probability distribution used to characterize the true correspondence between sample videos and sample audio event semantic descriptions. It's easy to understand that in the second matching probability, the matching probability between each sample video and its corresponding sample audio event semantic description is 1, while the matching probability between each sample video and other sample audio event semantic descriptions in the first training sample set is 0. For example, if the first training sample set includes 100 one-to-one corresponding sample videos and sample audio event semantic descriptions, and the 10th sample video corresponds to the 10th sample audio event semantic description, then in the second matching probability, the matching probability between the sample video feature corresponding to the 10th sample video and the sample audio event semantic description feature corresponding to the 10th sample audio event semantic description is 1, and the matching probability between the sample video feature corresponding to the 10th sample video and the sample audio event semantic description features of the other 99 sample audio event semantic descriptions in the first training sample set is 0. This allows us to determine the matching probability distribution corresponding to each sample video feature in the second matching probability; similarly, we can determine the matching probability distribution corresponding to each sample audio event semantic description feature in the second matching probability, thus determining the second matching probability.

[0224] In step 2640, after determining the first matching probability and the second matching probability, the cross-entropy between the first and second matching probabilities can be calculated as the first loss value. Cross-entropy measures the similarity between two probability distributions; the more similar the two probability distributions, the smaller the corresponding cross-entropy. By using cross-entropy as the first loss value, the current semantic feature generation model and / or the video feature generation model can learn how to encode text and video with similar semantics into similar feature representations. After training, an optimized semantic feature generation model can be obtained based on the current semantic feature generation model, and / or an optimized video feature generation model can be obtained based on the video feature generation model. In other words, through steps 2610-2640, the semantic feature generation model can be trained independently, or the semantic feature generation model and the optimized video feature generation model can be trained together; this application does not specifically limit this. Using the semantic feature generation model to encode the semantic description of sound events results in a high similarity between the encoded semantic features of the target sound event and the video features obtained by encoding the video representing the corresponding sound event. For example, if the semantic description of a sound event is "a person is hammering an object", then the target sound event semantic features obtained by encoding this semantic description using a trained semantic feature generation model will have a high degree of similarity to the video features obtained by encoding the video of "a person is hammering an object" using a video generation model.

[0225] In steps 2610 to 2640, by using a video feature generation model and a semantic feature generation model, the semantic descriptions of sample videos and sample sound events are encoded into corresponding sample video features and sample sound event semantic description features, respectively. Then, a first matching probability is predicted between the sample video and sample sound event semantic descriptions, and a second matching probability is determined based on the correspondence between them. The cross-entropy between the first and second matching probabilities is used as a first loss value to optimize the parameters of the video feature generation model and the semantic feature generation model. This allows the video feature generation model and the semantic feature generation model to encode semantically similar videos and texts into video features and text features with high similarity, respectively. Thus, by using the trained semantic feature generation model to encode the semantic descriptions of sound events, the resulting sound event semantic features have a high similarity to the video features obtained by encoding the video representing the corresponding sound event.

[0226] The training process of the feature adapter:

[0227] Referring to Figure 27, in one embodiment, the semantic feature adapter can be trained in the following manner:

[0228] Step 2710: Copy the encoding layer in the diffusion model to obtain the initialized semantic feature adapter, freeze the parameters of the diffusion model, and the initialized semantic feature adapter includes adapter layers corresponding to the multiple encoding layers of the diffusion model respectively.

[0229] Step 2720: Obtain the second training sample set, which includes at least one sample content;

[0230] Step 2730: Extract the original audio features, target sound event semantic features, and sample content features from the sample content;

[0231] Step 2740: Perform multiple noise-adding steps on the original audio features of the sample using a diffusion model to obtain noisy audio features, and store the forward noise added to the original audio features of the sample in each noise-adding step;

[0232] Step 2750: Input the sample content features into the diffusion model so that the diffusion model performs multiple denoising steps on the noisy audio features based on the sample content features; in each denoising step, input the semantic features of the sample target sound event into the initialized semantic feature adapter to obtain the sample adjustment features corresponding to each coding layer in the initialized semantic feature adapter, and adjust the output of the corresponding coding layer based on the sample adjustment features to obtain the inverse noise corresponding to a single denoising step, and multiple denoising steps and multiple noise addition steps correspond to each other.

[0233] Step 2760: Determine the second loss value based on the inverse noise and the forward noise of the corresponding noise-adding step, and optimize the parameters of each adapter layer based on the second loss value until the initialized semantic feature adapter satisfies the first condition, thus obtaining the semantic feature adapter.

[0234] In step 2710, during the initial training phase, an initialized semantic feature adapter can be obtained by copying the structure and parameters of each encoding layer of the diffusion model. This initialized semantic feature adapter is one that has not yet been fully trained. Specifically, referring to Figure 28, the diffusion model is based on the Unet network and includes encoding layers A to D, decoding layers A to D corresponding to encoding layers A to D, and an intermediate connection layer, which can also be considered an encoding layer. After copying the parameters of the diffusion model, adapter layers A to D and an intermediate adapter layer are obtained. These adapter layers (i.e., adapter layers A to D and the intermediate adapter layer) constitute the initialized semantic feature adapter. After copying the parameters of the diffusion model to obtain the initialized semantic feature adapter, the parameters of the diffusion model itself are frozen to prevent changes in the parameters during training. Since the initialized semantic feature adapter is obtained by copying the parameters and structure of multiple encoding layers of the diffusion model, the structure of the neural network layers in the initialized semantic feature adapter is consistent with the structure of multiple encoding layers in the diffusion model. The initialized semantic feature adapter will include adapter layers corresponding to each encoding layer. In addition, referring to Figure 28, a zero convolutional layer is set between the initialized semantic feature adapter and the diffusion model. This zero convolutional layer is a 1*1 convolutional layer. In the early stage of training, the weights and biases of the zero convolutional layer are set to 0. This can avoid the harmful noise generated from the semantic feature adapter from affecting the output of each coding layer of the diffusion model in the early stage of training. During the training process, the parameters of the adapter layer and the zero convolutional layer are optimized simultaneously.

[0235] In step 2720, after obtaining the initialized semantic feature adapter, a second training sample set for training the semantic feature adapter is acquired. The second training sample set includes multiple sample contents, which can be a sample video with sound effects. For example, the second training sample set can consist of two datasets: AudioSet and Vggsound.

[0236] Since sample content is often obtained from the internet, and the audio quality of online videos varies greatly, often containing a lot of noise, background noise, and other interfering sound effects, using only sample videos as sample content will result in the extracted original audio features containing a large amount of noise and background noise. When training the semantic feature adapter, it needs to learn how to predict the corresponding noise based on the given semantic features of the target sound event and the sample content features, and recover the original audio features from the noisy audio features. Because the original audio features themselves contain noise and background noise, the semantic feature adapter may incorrectly learn these noise and background noise-related components during training. Therefore, in one embodiment, in addition to sample videos, sample audio can also be used as sample content. This allows for the use of more high-quality, noise-free sample audio, reducing the noise and background noise components in the extracted original audio features and preventing the semantic feature adapter from incorrectly learning noise and background noise-related knowledge.

[0237] In step 2730, the original audio features of the sample are feature representations corresponding to the audio contained in the sample content. When the sample content is sample audio, the sample audio can be processed directly to obtain the corresponding feature representation; when the sample content is sample video with sound, the audio of the sample video with sound can be extracted and processed to obtain the corresponding feature representation. In one embodiment, Mel-spectral processing can be performed on the audio in the sample audio or sample video with sound to obtain Mel-spectral features, which can then be used as the corresponding original audio features of the sample.

[0238] The method for obtaining the semantic features of sample sound events can refer to the method for obtaining the semantic features of target sound events corresponding to the target video in the above embodiments. For example, for the semantic features of sample sound events, referring to Figure 29, multiple video frames can be extracted from the sample video with sound, and after adding corresponding preset prompts, they can be input into the visual understanding model to generate the corresponding semantic description of the sample sound event. Then, the semantic feature generation model trained on the sample sound event encodes the semantic description of the sample sound event into the corresponding semantic features of the sample target sound event. When the sample content is sample audio, the sample audio can be directly input into the corresponding audio language model to generate the corresponding sample audio description text.

[0239] Sample content features are used as input to the diffusion model to guide the diffusion process. For example, referring to Figure 29, when the sample content is a sample video with sound, the sample video with sound is encoded by a visual encoder to obtain the corresponding sample content features; when the sample content is a sample audio, the sample audio is encoded by an audio encoder to obtain sample audio features, which are used as sample content features.

[0240] It should be noted that the visual features of the samples are obtained by encoding the sample video with sound using a visual encoder, while the audio features of the samples are obtained by encoding the sample audio using an audio encoder. The visual encoder and the audio encoder are jointly trained. They can encode the time- and semantically aligned video and audio into visual features and audio features with high similarity and alignment in the time and semantic dimensions, respectively. The training process of the visual encoder and the audio encoder will be explained later and will not be elaborated here.

[0241] In step 2740, the original audio features of the sample are first subjected to multiple noise-adding steps using a diffusion model, thereby performing noise-adding processing on the original audio features of the sample to obtain noisy audio features. Simultaneously, the noise added to the original audio features of the sample by each diffusion model in each noise-adding step is recorded, i.e., the forward noise corresponding to each noise-adding step. Specifically, referring to Figure 29, the original audio features of the sample are first encoded by a variational autoencoder to obtain the corresponding feature representation. This feature representation is then input into the diffusion model. During the forward diffusion process, the diffusion model directly performs noise prediction based on this feature representation, generating the noise corresponding to each noise-adding step and adding it to the feature representation, thereby gradually transforming the original audio features of the sample into noisy audio features.

[0242] In step 2750, the process of performing multiple denoising steps on the noisy audio features based on the sample content features using a diffusion model for diffusion processing can be referred to in step 320 above, where the diffusion processing is performed on the audio features to be diffused based on visual features. Referring to Figure 29, for example, when the sample content is a sample video with sound, multiple keyframes can be extracted from the sample content and input into a visual language model to obtain the corresponding semantic description of the sample sound event. Then, the semantic description of the sample sound event is encoded through a semantic feature generation model to obtain the semantic features of the sample target sound event.

[0243] In each denoising step of the diffusion process, the semantic features of the target sound event are input into the semantic feature adapter. Referring to Figure 28, in the adapter, the semantic features of the target sound event are first processed by a zero-convolution layer. Simultaneously, the input of the first encoding layer (encoding layer A) of the multiple encoding layers in the diffusion model is obtained. The input of encoding layer A and the zero-convolution processed semantic features of the target sound event are superimposed to obtain the superimposed features. This superimposed feature is input into the first adapter layer (adapter layer A) of the semantic feature adapter, where corresponding processing is performed to obtain the sample adjustment features corresponding to adapter layer A. Furthermore, the multiple adapter layers in the semantic feature adapter are cascaded sequentially. The output of adapter layer A also serves as the input of adapter layer B. Adapter layer B processes the corresponding input to obtain the corresponding output, which is the sample adjustment feature corresponding to encoding layer B, and also the input of adaptation layer C. This process continues until the sample adjustment features corresponding to each encoding layer are obtained. Finally, the sample adjustment features are superimposed onto the output of the corresponding encoding layer.

[0244] It should be noted that in the diffusion model of the Unet architecture, the output of each encoding layer also serves as the input to the next encoding layer. The sample adjustment features are only superimposed on the output from the encoding layer to the corresponding decoding layer, and not on the output from the encoding layer to the next encoding layer. For example, in Figure 28, the sample adjustment features generated by the adapter layer A are only superimposed on the output from encoding layer A to decoding layer A, but not on the output from encoding layer A to encoding layer B.

[0245] In one embodiment, referring to FIG28, the output of each adapter layer is also connected to a zero convolutional layer. After obtaining the output of each adapter layer, the output of each adapter layer is connected to a zero convolutional layer. After the zero convolutional layer performs zero convolution processing on the output of the adapter layer, the sample adjustment features from the adapter layer input to the diffusion model are obtained.

[0246] In step 2760, the first condition is a preset condition for determining whether the initialized semantic feature adapter has been trained. For example, it can be comparing the loss value during training with a set loss threshold; or, the number of training iterations can be set, for example, after training the semantic feature adapter with 300 sample contents, the obtained semantic feature adapter is tested using a test set, and the performance of the semantic feature adapter is evaluated based on the test set to determine whether the semantic feature adapter has been trained.

[0247] Specifically, since each denoising step corresponds to a denoising step, the corresponding loss value, i.e., the second loss value, can be calculated by comparing the noise predicted in each denoising step with the noise added to the original audio features of the sample in the corresponding denoising step. Then, backpropagation is performed based on the second loss value. It should be noted that, since the parameters of the diffusion model itself are frozen, during backpropagation, based on the chain rule, only the gradients of each adapter layer in the initialized semantic feature adapter and the gradient of the zero-convolutional layer connecting the semantic feature adapter and the diffusion model are calculated, without calculating the gradients of the encoding layer of the diffusion model itself or other neural network layers. Afterwards, the zero-convolutional layer and each adapter layer perform gradient descent processing on the corresponding parameters based on their respective gradients, thereby optimizing the corresponding parameters until the initialized semantic feature adapter meets the set first condition. This indicates that the initialized semantic feature adapter has learned well how to generate the corresponding first adjustment features based on the semantic features of the input target sound event. At this point, the trained semantic feature adapter is obtained.

[0248] In the embodiments disclosed in steps 2710 to 2760, an initialized semantic feature adapter is obtained by copying the parameters of each coding layer of the diffusion model, while the parameters of the diffusion model are frozen. Then, corresponding sample content features, original sample audio features, and target sound event semantic features are extracted from the sample content. The original sample audio features are then denoised to obtain noisy audio features. Next, the sample content features are input into the diffusion model, causing the diffusion model to perform multiple denoising steps on the noisy audio features based on the sample content features. Simultaneously, in each denoising step, the target sound event semantic features are input into the semantic feature adapter, which performs corresponding processing on the target sound event semantic features to obtain sample adjustment features corresponding to each coding layer. This adjusts the output of each coding layer of the diffusion model during the diffusion process, thereby predicting the noise corresponding to the denoising step. Next, based on the noise predicted in the denoising step and the noise added to the original audio features of the sample in the corresponding noise addition step, a second loss value is calculated. Then, backpropagation is performed based on this second loss value to calculate the gradient for each adapter layer. The parameters of each adapter layer are optimized using gradient descent. The semantic feature adapter satisfies the first condition, thus learning how to process the semantic features of the target sound event into corresponding feature representations for injection into the diffusion process. Once the semantic feature adapter is trained, it can inject the semantic features of the target sound event into the diffusion process, enabling the diffusion model to generate accompanying audio that is semantically aligned with the target video based on the additional injected semantic features of the target sound event.

[0249] Steps 2710 to 2760 above illustrate the training process of the semantic feature adapter. It is understood that the training process for the timestamp feature adapter used to generate the second adjusted feature based on the sound event timestamp features can also refer to the training process for the semantic feature adapter in steps 2710 to 2760. The difference is that when training the timestamp feature adapter, the sample sound event timestamp features are extracted from the sample content as input to the timestamp feature adapter, without needing to extract the sample target sound event semantic features from the sample content. For the sample sound event timestamp features, since both the sample video and sample audio inherently carry audio signals, the occurrence time of the sound event can be determined by detecting sound events in the audio signals carried by the sample content, and this time point can be used as the corresponding sample sound event timestamp feature. For example, the ATST-Frame model (frame-level audio teacher-student transformer model) can be used to detect sound events in the audio signals carried by the sample content to determine the corresponding timestamp features of the sound events. Alternatively, the audio energy of the audio signals in the sample content can be detected to determine the corresponding timestamp features of the sample sound events. For instance, the librosa.onset.onset_detect module in the librosa toolkit can be used to detect the timestamp features of sample audio or sample video with sound. After obtaining the timestamp features of the sample sound events, they are input into the timestamp feature adapter to determine the corresponding second adjusted features of the sample. Based on the second adjusted features of the sample, the output of the coding layer of the diffusion model is adjusted so that the diffusion model predicts the corresponding noise based on the timestamp features of the sample sound events. The corresponding loss value is determined by combining the noise added to the original audio features of the sample in the corresponding noise addition step. Backpropagation is performed based on the loss value to determine the gradient of the timestamp feature adapter, and the parameters of the timestamp feature adapter are optimized by gradient descent until the timestamp feature adapter meets the first condition, resulting in a trained timestamp feature adapter.

[0250] Training processes for visual encoders and audio encoders:

[0251] When using sample audio as sample content, it needs to be encoded by an audio encoder to obtain corresponding sample audio features, which are then input into the diffusion model as sample content features. Since the diffusion model itself predicts noise based on the visual features of the video, when training the feature adapter using sample audio as sample content, it is necessary to ensure that the sample audio features obtained from the sample audio and the visual features obtained from the video corresponding to the sample audio are aligned semantically and temporally. This requires joint training of the visual encoder used to encode sample audio video into sample visual features and the audio encoder used to encode sample audio into corresponding sample audio features.

[0252] Referring to Figure 30, in one embodiment, the sample content is sample audio or sample video with sound. The sample content features include one of sample visual features and sample audio features. The sample visual features are obtained by encoding the sample video with sound using a visual encoder, and the sample audio features are obtained by encoding the sample video with sound or sample audio using an audio encoder. The visual encoder and audio encoder are jointly trained in advance in the following manner:

[0253] Step 3010: Obtain the third training sample set, wherein the third training sample set includes multiple sample audio-video pairs, and the sample audio-video pairs include corresponding training sample audio and training sample video.

[0254] Step 3020: Encode each training sample audio and each training sample video using the current audio encoder and the current visual encoder respectively to obtain multiple training sample audio features and multiple training sample visual features.

[0255] Step 3030: The training sample audio and training sample video in a single sample audio-video pair are segmented into multiple corresponding sample audio segments and multiple sample video segments respectively. The current audio encoder and the current visual encoder encode the multiple sample audio segments and multiple sample video segments respectively to obtain multiple sample segment audio features and multiple sample segment visual features.

[0256] Step 3040: Determine the semantic alignment score based on the audio features of multiple training samples and the visual features of multiple training samples, and determine the temporal alignment score based on the audio features of multiple sample segments and the visual features of multiple sample segments;

[0257] Step 3050: Determine the audio-video alignment score based on the semantic alignment score and the temporal alignment score, and optimize the current audio encoder and the current visual encoder based on the audio-video alignment score until the audio-video alignment score is greater than the alignment score threshold, thus obtaining the audio encoder and the visual encoder.

[0258] In step 3010, the third training sample set is a dataset containing multiple sample audio-video pairs. Each sample audio-video pair includes a corresponding training sample video and a training sample audio. The training sample video and training sample audio in a single sample audio-video pair are aligned in the temporal and semantic dimensions.

[0259] In step 3020, the current audio encoder is an audio encoder that has not yet been fully trained. It encodes each training sample audio to obtain corresponding training sample audio features. Similarly, the current visual encoder is a visual encoder that has not yet been fully trained. It encodes each training sample video to obtain multiple training sample visual features. The visual and audio features obtained from the training sample video and audio within the same audio-visual pair are mutually corresponding.

[0260] In step 3030, the training sample video and training sample audio in each sample audio-video pair are segmented into multiple sample audio segments and sample video segments, respectively. It is understood that the segmentation rules for the training sample video and training sample audio are consistent. For example, when segmenting the training sample video, each 8-second segment is taken as a sample video segment. Similarly, when segmenting the training sample audio, each 8-second segment is taken as an audio segment. The video segment consisting of the first 8 seconds of the training sample video corresponds to the audio segment consisting of the first 8 seconds of the training sample audio in the same audio-video pair. The video segment consisting of the 9th to 16th seconds of the training sample video corresponds to the audio segment consisting of the 9th to 16th seconds of the training sample audio in the same audio-video pair, and so on. This ensures a one-to-one correspondence between the segmented video and audio segments.

[0261] After segmenting the training sample video and training sample audio into multiple sample video segments and sample audio segments respectively, the current visual encoder encodes each sample video segment into sample segment visual features, and the current audio encoder encodes each sample audio segment into corresponding sample segment audio features.

[0262] In step 3040, after encoding the training sample video and training sample audio in each sample audio-video pair to obtain multiple training sample visual features and training sample audio features, the matching score between the training sample audio features and each training sample visual feature can be determined by calculating the cosine similarity between each training sample visual feature and training sample audio feature. Specifically, the matching score between the i-th training sample audio and the j-th training sample video can be expressed as:

[0263] Among them, L S (i,j) represents the matching score between the i-th training sample audio and the j-th training sample video, N S Let represent the number of audio / video pairs in the third training sample set, sim() represent the cosine similarity function, τ is a preset hyperparameter, and Eia E represents the training sample audio feature corresponding to the i-th training sample audio. jv E represents the visual features of the training sample corresponding to the j-th training sample video. kv E represents the visual features of the training sample corresponding to the k-th training sample video. ka Let L represent the training sample audio features corresponding to the k-th training sample audio. Then, based on the true matching probability between each training sample video and each training sample audio, the matching scores between each training sample video and each training sample audio are weighted to obtain the semantic alignment score. For example, the i-th training sample video and the i-th training sample audio correspond to L, therefore, L... S The score of (i,i) is multiplied by the weighting factor 1, and L is... S (i,j) j≠i Multiply by 0. This gives us the semantic alignment score, denoted as L. S .

[0264] After determining the semantic alignment score, since audio and video need to be aligned not only semantically but also temporally, the temporal alignment score can be calculated by segmenting a single audio-video pair into multiple video and audio segments. Specifically, the matching score between each video and audio segment in a single audio-video pair can be determined by calculating the cosine similarity between the visual and audio features of the video and audio segments. The matching score between the video and audio segments can be determined using the following formula:

[0265] Among them, L T (i,j) represents the matching score between the i-th audio sample and the j-th video sample, N T This represents the number of sample video or audio segments obtained from a single sample audio-video pair, sim() represents the cosine similarity function, τ is a preset hyperparameter, and e ia e represents the audio feature of the sample segment corresponding to the i-th sample audio segment. jv Let e ​​represent the visual features of the sample segment corresponding to the j-th sample video segment. kv e represents the visual features of the sample segment corresponding to the k-th sample video segment. ka Let L represent the audio features of the sample segment corresponding to the k-th sample audio segment. Based on the true correspondence between each sample video segment and each sample audio segment corresponding to a single sample audio-video pair, the matching scores between each sample video segment and each sample audio segment are weighted to obtain a time alignment score. For example, the i-th sample video segment and the i-th sample audio segment correspond to L... TThe score of (i,i) is multiplied by the weighting factor 1, and L is... T (i,j) j≠i Multiply by 0. This gives you the time alignment score, denoted as L. T .

[0266] In step 3050, the audio-video alignment score is the sum of the weighted scores of temporal alignment and semantic alignment; the alignment score threshold is a preset threshold. Specifically, after determining the temporal alignment score and semantic alignment score, the audio-video alignment score L = L0 can be determined using the following formula. S +λL T Where L represents the audio-video alignment score, and λ is a preset coefficient, which is set to 1 in this embodiment. The audio-video alignment score characterizes the alignment degree between the audio features of the training samples and audio features of the sample segments obtained by the current audio encoder and the visual features of the training samples and visual features of the sample segments obtained by the current visual encoder in the temporal and semantic dimensions. By comparing the audio-video alignment score with an alignment score threshold, the current audio encoder and the current visual encoder are optimized to maximize the audio-video alignment score. When the audio-video alignment score is greater than the alignment score threshold, it indicates that the current audio encoder and the current visual encoder can encode the audio and video that are aligned in the temporal and semantic dimensions into audio features and visual features with high similarity, respectively. Thus, a trained audio encoder and a trained visual encoder are obtained based on the current audio encoder.

[0267] In the embodiments disclosed in steps 3010 to 3050, a visual encoder and an audio encoder are trained using multiple sample audio-video pairs. This allows the audio encoder and visual encoder to encode audio and video that are aligned in the temporal and semantic dimensions into audio features and visual features with high similarity, respectively. In this way, sample audio can be used as sample content, and the audio encoder can encode the sample audio into corresponding sample audio features to train the feature adapter. This allows the feature adapter to be trained using more high-quality audio without noise or background noise, enabling the feature adapter to learn how to accurately encode the semantic features or timestamp features of the target sound event into the corresponding first adjustment feature or second adjustment feature. This accurately adjusts the first output of each encoding layer of the diffusion model, generating accompanying audio that is semantically and temporally aligned with the target video.

[0268] Example of the specific usage process of the accompanying audio generation method in this embodiment:

[0269] Referring to FIG31, a specific usage process of the accompanying audio generation method of this disclosure embodiment is described in detail below. The process includes, but is not limited to, the following steps 3101 to 3122. In this process, the server 140 and the object terminal 110 each undertake a part of the task of the accompanying audio generation method. In actual use, the server 140 may also undertake all the tasks of the accompanying audio generation method.

[0270] 3101. Server 140 extracts visual features from the target video;

[0271] 3102. Server 140 decomposes the target video into video frames;

[0272] 3103. Server 140 inputs the decomposed video frames and preset prompts into the visual language model so that the visual language model can generate a semantic description of the sound event based on the video frames;

[0273] 3104. Server 140 encodes the semantic description of the sound event through a semantic feature generation model to obtain the semantic features of the target sound event;

[0274] 3105. The target terminal 110 obtains the positive prompt word and the negative prompt word, and sends the positive prompt word and the negative prompt word to the server 140;

[0275] 3106. The server 140 encodes the positive and negative prompts received from the target terminal 110 into corresponding prompt features, and concatenates the prompt features to the target sound event semantic features to update the target sound event semantic features;

[0276] 3107. The object terminal 110 obtains the first number and sends the first number to the server 140;

[0277] 3108. Service 140 generates a first number of random seeds based on a first number received from object terminal 110, and generates a first number of audio features to be diffused based on the first number of random seeds;

[0278] 3109. Server 140 inputs a first number of audio features to be diffused into the diffusion model, and performs diffusion processing on the first number of audio features to be diffused based on visual features;

[0279] 3110. Server 140 obtains the initial input of the first coding layer of the diffusion model;

[0280] 3111. Server 140 performs zero-convolution processing on the semantic features of the target sound event, and then superimposes the semantic features of the target sound event after zero-convolution processing with the initial input to obtain the superimposed semantic features of the target sound event;

[0281] 3112. Server 140 inputs the superimposed semantic features of the target sound event into the corresponding semantic feature adapter, and performs zero convolution on the output of the semantic feature adapter to obtain the first adjusted feature;

[0282] 3113. The target terminal 110 obtains the first operation on the target video and sends the first operation to the server 140;

[0283] 3114. Server 140 determines multiple first annotation information based on the first operation received from object terminal 110, and determines sound event timestamp features based on the multiple first annotation information;

[0284] 3115. Server 140 performs sound event detection based on the visual features of the target video to determine at least one sound event in the target video, so as to determine at least one time node corresponding to the at least one sound event in the target video;

[0285] 3116. Server 140 determines the timestamp characteristics of the sound event based on at least one time node;

[0286] 3117. Server 140 generates a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature;

[0287] 3118. In the coding layer, server 140 determines the intermediate features of the audio to be diffused based on the features of the audio to be diffused;

[0288] 3119. Server 140 generates a query matrix based on intermediate features of the audio to be diffused, and generates a key matrix and a value matrix based on visual features;

[0289] 3120. Server 140 performs cross-attention processing based on the query matrix, key matrix, and value matrix, and determines the first output corresponding to the encoding layer based on the result of the cross-attention processing;

[0290] 3121. The object terminal 110 obtains the first weight coefficient corresponding to the first adjustment feature and the second weight coefficient corresponding to the second adjustment feature, and sends the first weight coefficient and the second weight coefficient to the server 140;

[0291] 3122. Server 140 performs weighted processing on the first adjustment feature and the second adjustment feature based on the first weight coefficient and the second weight coefficient received from object terminal 110, respectively, to obtain the weighted first adjustment feature and the weighted second adjustment feature, and accumulates the weighted first adjustment feature and the weighted second adjustment feature to the first output of the corresponding coding layer to obtain the third output;

[0292] 3123. Server 140 generates accompanying audio for the target video based on the third output, such that the accompanying audio contains audio segments corresponding to the semantic features of the target sound event according to the timestamps corresponding to the sound event timestamp features.

[0293] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0294] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0295] Description of apparatus and devices according to embodiments of this disclosure:

[0296] Referring to Figure 32, which is a schematic diagram of the structure of the accompanying audio generation apparatus 3200 provided in an embodiment of the present disclosure, the accompanying audio generation apparatus 3200 includes:

[0297] The feature extraction unit 3210 is used to extract the visual features of the target video and obtain the semantic features of the target sound events in the target video.

[0298] The diffusion execution unit 3220 is used to input visual features into a diffusion model, the diffusion model including an encoding layer for diffusion processing of audio features to be diffused based on the visual features;

[0299] The diffusion adjustment unit 3230 is used to generate a first adjustment feature corresponding to the coding layer based on the semantic features of the target sound event, adjust the first output of the coding layer based on the first adjustment feature to obtain a second output, and generate the accompanying audio of the target video based on the second output, so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event.

[0300] Optionally, the feature extraction unit 3210 is used for:

[0301] Identify sound events from the target video and obtain the target frames containing the sound events;

[0302] The target frame and the context frame of the target frame in the target video are input into the importance prediction model, and the importance of the sound event is predicted by the importance prediction model.

[0303] Get the sound intensity corresponding to the sound event;

[0304] Based on importance and sound intensity, the target sound event is determined from the identified sound events;

[0305] Obtain the semantic features of the target sound event.

[0306] Optionally, the feature extraction unit 3210 is used for:

[0307] Decompose the target video into video frames;

[0308] The decomposed video frames and preset prompts are input into the visual language model so that the visual language model can generate semantic descriptions of sound events based on the video frames. The preset prompts are used to prompt the visual language model to generate semantic descriptions of sound events based on the video frames.

[0309] The semantic description of the sound event is encoded by a semantic feature generation model to obtain the semantic features of the target sound event.

[0310] Optionally, the accompanying audio generation device 3200 further includes a first training unit (not shown), the first training unit being used for:

[0311] Obtain the first training sample set, which includes multiple video-text pairs, and each video-text pair includes corresponding sample video and sample sound event semantic descriptions.

[0312] Each sample video is encoded using a video feature generation model, and the semantic description of each sample sound event is encoded using a current semantic feature generation model, resulting in multiple sample video features and sample sound event semantic description features.

[0313] Predict the first matching probability between each sample video feature and each sample sound event semantic description feature, and determine the second matching probability between each sample video feature and each sample sound event semantic description feature based on the correspondence between the sample video and the sample sound event semantic description.

[0314] The first loss value is determined based on the first matching probability and the second matching probability. The current semantic feature generation model is then optimized based on the first loss value until the first loss value is less than the first loss threshold, thus obtaining the semantic feature generation model.

[0315] Optionally, the feature extraction unit 3210 is used for:

[0316] Obtain the cue word set and encode the cue word set into corresponding cue word features; wherein, the cue word set includes positive cue words and negative cue words, positive cue words are used to indicate sound events contained in the accompanying audio generated by the diffusion model, and negative cue words are used to indicate sound events not contained in the accompanying audio generated by the diffusion model;

[0317] The cue word features are concatenated to the target sound event semantic features to update the target sound event semantic features.

[0318] Optionally, the diffusion execution unit 3220 is used for:

[0319] In the coding layer, intermediate features of the audio to be diffused are determined based on the features of the audio to be diffused;

[0320] A query matrix is ​​generated based on intermediate features of the audio to be diffused, and a key matrix and a value matrix are generated based on visual features.

[0321] Cross-attention processing is performed based on the query matrix, key matrix, and value matrix, and the first output corresponding to the encoding layer is determined based on the result of the cross-attention processing.

[0322] Optionally, the diffusion execution unit 3220 is used for:

[0323] Obtain a first number, generate a first number of random seeds, and generate a first number of audio features to be diffused based on the first number of random seeds;

[0324] The first number of audio features to be diffused are input into the diffusion model, and diffusion processing is performed on the first number of audio features to be diffused based on visual features to obtain the first number of candidate accompanying audio.

[0325] The accompanying audio is determined from the first number of candidate accompanying audios.

[0326] Optionally, the diffusion adjustment unit 3230 is used for:

[0327] Obtain the initial input of the first coding layer of the diffusion model;

[0328] The semantic features of the target sound event are subjected to zero convolution processing. The semantic features of the target sound event after zero convolution processing are superimposed with the initial input to obtain the superimposed semantic features of the target sound event.

[0329] The semantic features of the superimposed target sound event are input into the corresponding semantic feature adapter, and the output of the semantic feature adapter is subjected to zero convolution to obtain the first adjusted feature.

[0330] Optionally, the accompanying audio generation device 3200 further includes a second training unit (not shown), the second training unit being used for:

[0331] The encoding layers in the diffusion model are copied to obtain the initialized semantic feature adapter, and the parameters of the diffusion model are frozen. The initialized semantic feature adapter includes adapter layers corresponding to the multiple encoding layers of the diffusion model.

[0332] Obtain a second training sample set, which includes at least one sample content;

[0333] Extract original audio features, semantic features of target sound events, and content features from the sample content;

[0334] The original audio features of the sample are subjected to multiple noise-adding steps by a diffusion model to obtain noisy audio features. The forward noise added to the original audio features of the sample in each noise-adding step is stored.

[0335] The sample content features are input into the diffusion model so that the diffusion model performs multiple denoising steps on the noisy audio features based on the sample content features. In each denoising step, the semantic features of the sample target sound event are input into the initialized semantic feature adapter to obtain the sample adjustment features corresponding to each coding layer in the initialized semantic feature adapter. The output of the corresponding coding layer is adjusted based on the sample adjustment features to obtain the inverse noise corresponding to a single denoising step. Multiple denoising steps correspond to multiple noise addition steps.

[0336] The second loss value is determined based on the inverse noise and the forward noise of the corresponding noise-adding step, and the parameters of each adapter layer are optimized based on the second loss value until the initialized semantic feature adapter satisfies the first condition, thus obtaining the semantic feature adapter.

[0337] Optionally, the accompanying audio generation device 3200 also includes a third training unit (not shown), which is used for:

[0338] Obtain the third training sample set, which includes multiple audio-visual pairs, each of which includes the corresponding training sample audio and training sample video.

[0339] The audio and video of each training sample are encoded by the current audio encoder and the current visual encoder respectively, resulting in multiple audio features and multiple visual features of the training samples.

[0340] The training sample audio and training sample video in a single sample audio-video pair are segmented into multiple corresponding sample audio segments and multiple sample video segments, respectively. The current audio encoder and the current visual encoder encode the multiple sample audio segments and multiple sample video segments respectively to obtain the audio features and visual features of multiple sample segments.

[0341] Semantic alignment scores are determined based on audio features and visual features of multiple training samples, and temporal alignment scores are determined based on audio features and visual features of multiple sample segments.

[0342] The audio and video alignment scores are determined based on the semantic alignment score and the temporal alignment score. The current audio encoder and the current visual encoder are optimized based on the audio and video alignment scores until the audio and video alignment scores are greater than the alignment score threshold, thus obtaining the audio encoder and the visual encoder.

[0343] Optionally, the feature extraction unit 3210 is used for:

[0344] Obtain the timestamp features of audio events in the target video;

[0345] The diffusion adjustment unit 3230 is used for:

[0346] A second adjustment feature corresponding to the coding layer is generated based on the timestamp features of sound events;

[0347] The second output is adjusted based on the second adjustment feature to obtain the third output;

[0348] The accompanying audio of the target video is generated based on the third output, so that the accompanying audio contains audio segments corresponding to the semantic features of the target sound event according to the timestamps corresponding to the sound event timestamp features.

[0349] Optionally, the feature extraction unit 3210 is used for:

[0350] Identify target audio events from the target video;

[0351] Get the duration of the target sound event;

[0352] Obtain the event type of the target sound event;

[0353] If the duration and event type meet the predetermined conditions, obtain the audio event timestamp features of the target video.

[0354] Optionally, the feature extraction unit 3210 is used for:

[0355] If the target sound event is an object speech event, identify the mouth size of the object in the frame corresponding to the object speech event in the target video;

[0356] If the mouth size of the object exceeds the predetermined size, obtain the audio event timestamp features of the target video.

[0357] Optionally, the feature extraction unit 3210 is used for:

[0358] In response to the first operation of the target terminal on the target video, multiple first annotation information are determined, wherein the first annotation information is used to annotate the time nodes of the sound events in the target video;

[0359] The timestamp features of the sound event are determined based on multiple first annotation information.

[0360] Optionally, the feature extraction unit 3210 is used for:

[0361] Sound event detection is performed based on the visual features of the target video to determine at least one sound event in the target video, and to determine at least one time node corresponding to the at least one sound event in the target video.

[0362] The timestamp feature of a sound event is determined based on at least one time point.

[0363] Optionally, the diffusion adjustment unit 3230 is used for:

[0364] Obtain the first weight coefficient corresponding to the first adjustment feature and the second weight coefficient corresponding to the second adjustment feature, wherein the sum of the first weight coefficient and the second weight coefficient is not greater than the first threshold.

[0365] The first adjusted feature and the second adjusted feature are weighted based on the first weight coefficient and the second weight coefficient, respectively, to obtain the weighted first adjusted feature and the weighted second adjusted feature. The weighted first adjusted feature and the weighted second adjusted feature are accumulated to the first output of the corresponding coding layer to obtain the third output.

[0366] Referring to Figure 33, which is a structural block diagram of a portion of a target terminal implementing the accompanying audio generation method of this disclosure, the target terminal includes: a radio frequency (RF) circuit 3310, a memory 3315, an input unit 3330, a display unit 3340, a sensor 3350, an audio circuit 3360, a wireless fidelity (WiFi) module 3370, a processor 3380, and a power supply 3390, etc. Those skilled in the art will understand that the target terminal structure shown in Figure 33 does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0367] The RF circuit 3310 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 3380; in addition, it transmits uplink data to the base station.

[0368] The memory 3315 can be used to store software programs and modules. The processor 3380 executes various functional applications and data processing of the content object terminal by running the software programs and modules stored in the memory 3315.

[0369] The input unit 3330 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content object terminal. Specifically, the input unit 3330 may include a touch panel 3331 and other input devices 3332.

[0370] The display unit 3340 can be used to display input or provided information, as well as various menus of the content object terminal. The display unit 3340 may include a display panel 3341.

[0371] Audio circuit 3360, speaker 3361, and microphone 3362 provide an audio interface.

[0372] In this embodiment, the processor 3380 included in the object terminal can execute the accompanying audio generation method of the previous embodiment.

[0373] The target terminals of this disclosure include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to automatic image matching and text-based video dubbing.

[0374] Figure 34 is a partial structural block diagram of a server 140 implementing the accompanying audio generation method of this disclosure. The server 140 can vary considerably due to different configurations or performance, and may include one or more central processing units (CPUs) 3422 (e.g., one or more processors) and memory 3432, and one or more storage media 3430 (e.g., one or more mass storage devices) storing application programs 3442 or data 3444. The memory 3432 and storage media 3430 may be temporary or persistent storage. The program stored in the storage media 3430 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server 140. Furthermore, the CPU 3422 may be configured to communicate with the storage media 3430 and execute the series of instruction operations in the storage media 3430 on the server 140.

[0375] Server 140 may also include one or more power supplies 3426, one or more wired or wireless network interfaces 3450, one or more input / output interfaces 3458, and / or one or more operating systems 3441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0376] The central processing unit 3422 in server 140 can be used to execute the accompanying audio generation method of the present disclosure embodiments.

[0377] This disclosure also provides a computer-readable storage medium for storing program code for executing the accompanying audio generation methods of the foregoing embodiments.

[0378] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the aforementioned accompanying audio generation method.

[0379] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0380] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0381] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0382] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0383] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0384] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0385] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server 140, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0386] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0387] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

A method for generating accompanying audio, the method being performed by a computer device, the method comprising: Extract the visual features of the target video and obtain the semantic features of the target sound events in the target video; The visual features are input into a diffusion model, which includes an encoding layer for performing diffusion processing on the audio features to be diffused based on the visual features. Based on the semantic features of the target sound event, a first adjustment feature corresponding to the coding layer is generated. Based on the first adjustment feature, the first output of the coding layer is adjusted to obtain a second output. Based on the second output, the accompanying audio of the target video is generated so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event. According to the accompanying audio generation method of claim 1, obtaining the semantic features of the target sound event in the target video includes: Identify sound events from the target video and obtain target frames containing the sound events; The target frame and the context frame of the target frame in the target video are input into the importance prediction model, and the importance of the sound event is predicted by the importance prediction model. Obtain the sound intensity corresponding to the sound event; Based on the importance and the sound intensity, the target sound event is determined from the identified sound events; Obtain the semantic features of the target sound event. According to the accompanying audio generation method of claim 1, obtaining the semantic features of the target sound event in the target video includes: The target video is decomposed into video frames; The decomposed video frames and preset prompts are input into a visual language model so that the visual language model generates a semantic description of the sound event based on the video frames. The preset prompts are used to prompt the visual language model to generate the semantic description of the sound event based on the video frames. The semantic description of the sound event is encoded by a semantic feature generation model to obtain the semantic features of the target sound event. According to claim 3, the accompanying audio generation method, the semantic feature generation model is pre-trained in the following manner: Obtain the first training sample set, where, The first training sample set includes multiple video-text pairs, and each video-text pair includes corresponding sample video and sample sound event semantic descriptions; Each of the sample videos is encoded using a video feature generation model, and the semantic description of each of the sample sound events is encoded using a current semantic feature generation model, resulting in multiple sample video features and sample sound event semantic description features. Predict a first matching probability between each of the sample video features and each of the sample sound event semantic description features, and determine a second matching probability between each of the sample video features and each of the sample sound event semantic description features based on the correspondence between the sample video and the sample sound event semantic description; A first loss value is determined based on the first matching probability and the second matching probability. The current semantic feature generation model is optimized based on the first loss value until the first loss value is less than the first loss threshold, thereby obtaining the semantic feature generation model. According to the accompanying audio generation method of claim 3 or 4, after encoding the semantic description of the sound event through a semantic feature generation model to obtain the semantic features of the target sound event, the method further includes: Obtain a set of prompt words and encode the set of prompt words into corresponding prompt word features; wherein, the set of prompt words includes positive prompt words and negative prompt words, the positive prompt words are used to prompt the accompanying audio generated by the diffusion model to correspond to a first reference sound event, and the negative prompt words are used to prompt the accompanying audio generated by the diffusion model to correspond to a second reference sound event; The prompt word features are concatenated to the target sound event semantic features to update the target sound event semantic features. According to any one of claims 1 to 5, the method for generating accompanying audio, wherein the diffusion processing of the audio features to be diffused based on the visual features includes: In the coding layer, intermediate features of the audio to be diffused are determined based on the audio features to be diffused; A query matrix is ​​generated based on the intermediate features of the audio to be diffused, and a key matrix and a value matrix are generated based on the visual features. Cross-attention processing is performed based on the query matrix, the key matrix, and the value matrix, and the first output corresponding to the encoding layer is determined based on the result of the cross-attention processing. The accompanying audio generation method according to any one of claims 1 to 6, the method further comprising: Obtain a first number, generate a first number of random seeds, and generate a first number of audio features to be diffused based on the first number of random seeds; The first number of audio features to be diffused are respectively input into the diffusion model, and diffusion processing is performed on the first number of audio features to be diffused based on the visual features to obtain a first number of candidate accompanying audio; the accompanying audio is determined from the first number of candidate accompanying audio. According to any one of claims 1 to 7, the method for generating accompanying audio, wherein generating a first adjustment feature corresponding to the coding layer based on the semantic features of the target sound event includes: Obtain the initial input of the first coding layer of the diffusion model; The semantic features of the target sound event are subjected to zero convolution processing, and the semantic features of the target sound event after zero convolution processing are superimposed with the initial input to obtain the superimposed semantic features of the target sound event. The superimposed semantic features of the target sound event are input into the corresponding semantic feature adapter, and the output of the semantic feature adapter is subjected to zero convolution to obtain the first adjusted feature. According to claim 8, the training process of the semantic feature adapter in the accompanying audio generation method includes: The encoding layer in the diffusion model is copied to obtain the initialized semantic feature adapter, which is then frozen. The parameters of the diffusion model, and the initialized semantic feature adapter includes adapter layers corresponding to the multiple coding layers of the diffusion model respectively; Obtain a second training sample set, which includes at least one sample content; Extract the original audio features, target sound event semantic features, and sample content features from the sample content; The original audio features of the sample are subjected to multiple noise-adding steps through the diffusion model to obtain noisy audio features, and the forward noise added to the original audio features of the sample in each noise-adding step is stored. The sample content features are input into the diffusion model, so that the diffusion model performs multiple denoising steps on the noisy audio features based on the sample content features; in each denoising step, the semantic features of the sample target sound event are input into the initialized semantic feature adapter to obtain the sample adjustment features corresponding to each coding layer in the initialized semantic feature adapter, and the output of the corresponding coding layer is adjusted based on the sample adjustment features to obtain the inverse noise corresponding to a single denoising step, and the multiple denoising steps correspond to the multiple noise addition steps; A second loss value is determined based on the inverse noise and the corresponding forward noise of the noise-adding step. The parameters of each adapter layer are optimized based on the second loss value until the initialized semantic feature adapter satisfies the first condition, thus obtaining the semantic feature adapter. The accompanying audio generation method according to claim 9, wherein, The sample content is sample audio or sample video with sound. The sample content features include one of sample visual features and sample audio features. The sample visual features are obtained by encoding the sample video with sound using a visual encoder. The sample audio features are obtained by encoding the sample video with sound or the sample audio using an audio encoder. The visual encoder and the audio encoder are jointly trained in advance in the following manner: Obtain a third training sample set, wherein the third training sample set includes multiple sample audio-video pairs, and the sample audio-video pairs include corresponding training sample audio and training sample video; The current audio encoder and the current visual encoder encode each training sample audio and each training sample video respectively to obtain multiple training sample audio features and multiple training sample visual features. The training sample audio and training sample video in a single sample audio-video pair are respectively segmented into multiple corresponding sample audio segments and multiple sample video segments. The current audio encoder and the current visual encoder respectively encode the multiple sample audio segments and the multiple sample video segments to obtain multiple sample segment audio features and multiple sample segment visual features. A semantic alignment score is determined based on the audio features and visual features of the multiple training samples, and a temporal alignment score is determined based on the audio features and visual features of the multiple sample segments. Based on the semantic alignment score and the temporal alignment score, an audio-video alignment score is determined. The current audio encoder and the current visual encoder are then optimized based on the audio-video alignment score until the audio-video alignment score is greater than the alignment score threshold, thus obtaining the audio encoder and the visual encoder. According to any one of claims 1 to 10, the accompanying audio generation method, after obtaining the semantic features of the target sound event in the target video, further includes: Obtain the audio event timestamp features of the target video; The step of generating accompanying audio for the target video based on the second output, such that the accompanying audio includes audio segments corresponding to the semantic features of the target sound event, includes: A second adjustment feature corresponding to the coding layer is generated based on the sound event timestamp feature; The second output is adjusted based on the second adjustment feature to obtain the third output; The accompanying audio of the target video is generated based on the third output, so that the accompanying audio includes audio segments corresponding to the semantic features of the target sound event according to the timestamps corresponding to the sound event timestamp features. According to the accompanying audio generation method of claim 11, obtaining the sound event timestamp features of the target video includes: Identify target sound events from the target video; Obtain the duration of the target sound event; Obtain the event type of the target sound event; If the duration and the event type meet predetermined conditions, obtain the audio event timestamp features of the target video. According to the accompanying audio generation method of claim 12, after identifying the target sound event from the target video, the method further includes: If the target sound event is an object speech event, identify the mouth size of the object in the frame corresponding to the object speech event in the target video; If the mouth size of the object exceeds a predetermined size, obtain the audio event timestamp features of the target video. According to any one of claims 11 to 13, the method for generating accompanying audio, wherein obtaining the sound event timestamp features of the target video includes: In response to a first operation of the target terminal on the target video, a plurality of first annotation information is determined, wherein the first annotation information is used to annotate the time nodes in the target video where sound events occur; The timestamp feature of the sound event is determined based on the multiple first annotation information. According to any one of claims 11 to 14, the method for generating accompanying audio, wherein obtaining the sound event timestamp features of the target video includes: Sound event detection is performed based on the visual features of the target video to determine at least one sound event in the target video, and to determine at least one time node corresponding to the at least one sound event in the target video. The timestamp feature of the sound event is determined based on the at least one time node. The accompanying audio generation method according to any one of claims 11 to 15, wherein adjusting the second output based on the second adjustment feature to obtain the third output includes: Obtain a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, wherein the sum of the first weight coefficient and the second weight coefficient is not greater than a first threshold. The first adjustment feature and the second adjustment feature are weighted based on the first weight coefficient and the second weight coefficient, respectively, to obtain the weighted first adjustment feature and the weighted second adjustment feature. The weighted first adjustment feature and the weighted second adjustment feature are then accumulated to the first output of the corresponding coding layer to obtain the third output. An accompanying audio generation device, comprising: The feature extraction unit is used to extract the visual features of the target video and obtain the semantic features of the target sound events in the target video. A diffusion execution unit is used to input the visual features into a diffusion model, the diffusion model including an encoding layer for performing diffusion processing on the audio features to be diffused according to the visual features; The diffusion adjustment unit is used to generate a first adjustment feature corresponding to the coding layer based on the semantic features of the target sound event, adjust the first output of the coding layer based on the first adjustment feature to obtain a second output, and generate the accompanying audio of the target video based on the second output, so that the accompanying audio includes an audio segment corresponding to the semantic features of the target sound event. An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the accompanying audio generation method according to any one of claims 1 to 16. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the accompanying audio generation method according to any one of claims 1 to 16. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the accompanying audio generation method according to any one of claims 1 to 16.

Images