Accompanying audio generation method, related device and medium
By extracting the semantic features of visual and sound events in videos and using a diffusion model to generate accompanying audio that is semantically aligned with the video, the problem of audio-video alignment in existing technologies is solved, thus improving the efficiency of video creation.
Patent Information
- Application Number
- CN202411217392.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-09-02
AI Technical Summary
In existing technologies, artificial intelligence models struggle to accurately capture semantic information in video features, resulting in misalignment between the generated audio and video, and wasting a significant amount of time for video creators to add voiceovers.
By extracting the visual features of the target video and the semantic features of the target sound event, diffusion processing is performed using the coding layer of the diffusion model to generate accompanying audio that is semantically aligned with the video.
It achieves semantically accurate alignment between the generated accompanying audio and video, reducing the time and effort required for video creators to add voiceovers.
Smart Images

Figure CN118737121B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of audio generation, and in particular to a method for generating accompanying audio, a related device and a medium. Background Art
[0002] Video dubbing is an important part of the video creation process. Traditional video dubbing requires a lot of time and energy of video creators. Especially in the creation process of self-media videos, video images can be assisted by a large number of video creation tools, and even automatically generated by artificial intelligence models. However, the videos generated by artificial intelligence models are often silent videos, and video creators also need to dub these silent videos, which leads to most of the time and energy of video creators being spent on video dubbing.
[0003] In the prior art, there are some methods that automatically generate accompanying audio corresponding to a video through an artificial intelligence model. These methods often encode the video into corresponding video features, and then input the video features into an audio generation model so that the audio generation model generates the corresponding audio. It is difficult for the audio generation model to capture all the semantics contained in the video features, resulting in the generated audio being semantically misaligned with the video. Summary of the invention
[0004] The embodiments of the present disclosure provide a method for generating accompanying audio, a related device, and a medium, which can enable the audio generated based on a video to be accurately aligned with the video in terms of semantics.
[0005] According to one aspect of the present disclosure, there is provided a method for generating accompanying audio, comprising:
[0006] Extracting visual features of a target video and obtaining semantic features of a target sound event of the target video;
[0007] inputting the visual features into a diffusion model, the diffusion model comprising an encoding layer for diffusively processing diffuse audio features according to the visual features;
[0008] A first adjustment feature corresponding to the coding layer is generated based on the semantic feature of the target sound event, and a first output of the coding layer is adjusted based on the first adjustment feature to obtain a second output, and an accompanying audio of the target video is generated based on the second output so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event.
[0009] According to one aspect of the present disclosure, there is provided an accompanying audio generating device, comprising:
[0010] A feature extraction unit, used to extract visual features of a target video and obtain semantic features of a target sound event of the target video;
[0011] a diffusion execution unit, configured to input the visual features into a diffusion model, the diffusion model comprising a coding layer for performing diffusion processing on diffuse audio features according to the visual features;
[0012] A diffusion adjustment unit is used to generate a first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event, and adjust the first output of the coding layer based on the first adjustment feature to obtain a second output, and generate accompanying audio of the target video based on the second output so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event.
[0013] Optionally, the feature extraction unit is used to:
[0014] Identifying a sound event from the target video and acquiring a target frame containing the sound event;
[0015] Inputting the target frame and the context frame of the target frame in the target video into a sound event importance prediction model to obtain a predicted importance of the sound event;
[0016] Obtaining the sound intensity corresponding to the sound event;
[0017] Determining a target sound event among the identified sound events based on the importance and the sound intensity;
[0018] Acquire the target sound event semantic feature of the target sound event.
[0019] Optionally, the feature extraction unit is used to:
[0020] Decomposing the target video into video frames;
[0021] Inputting the decomposed video frames and preset prompts into a visual language model, so that the visual language model generates the semantic description of the sound event based on the video frames, wherein the preset prompts are used to prompt the visual language model to generate the semantic description of the sound event based on the video frames;
[0022] The semantic description of the sound event is encoded through a semantic feature generation model to obtain the semantic features of the target sound event.
[0023] Optionally, the accompanying audio generating device further includes a first training unit, and the first training unit is used to:
[0024] Acquire a first training sample set, wherein the first training sample set includes a plurality of video-text pairs, and a single video-text pair includes a sample video and a sample sound event semantic description corresponding to each other;
[0025] Encoding each of the sample videos through the video feature generation model, and encoding each of the sample sound event semantic descriptions through the semantic feature generation model, to obtain a plurality of sample video features and sample sound event semantic description features;
[0026] Predicting a first matching probability between each of the sample video features and each of the sample sound event semantic description features, and determining a second matching probability between each of the sample video features and each of the sample sound event semantic description features based on a corresponding relationship between the sample video and the sample sound event semantic description;
[0027] A first loss value is determined based on the first matching probability and the second matching probability, and the semantic feature generation model and the video feature generation model are optimized based on the first loss value until the first loss value is less than a first loss threshold, thereby obtaining the semantic feature generation model.
[0028] Optionally, the feature extraction unit is used to:
[0029] Acquire a positive prompt word and a reverse prompt word, and encode the positive prompt word and the reverse prompt word into corresponding prompt word features; wherein the positive prompt word is used to prompt the diffusion model to correspond to a first reference sound event in the generated accompanying audio, and the reverse prompt word is used to prompt the diffusion model to correspond to a second reference sound event in the generated accompanying audio;
[0030] The prompt word feature is spliced to the target sound event semantic feature to update the target sound event semantic feature.
[0031] Optionally, the diffusion execution unit is used to:
[0032] In the encoding layer, determining an intermediate feature of the audio to be diffused based on the feature of the audio to be diffused;
[0033] Generate a query matrix based on the intermediate features of the audio to be diffused, and generate a key matrix and a value matrix based on the visual features;
[0034] Cross-attention processing is performed based on the query matrix, the key matrix and the value matrix, and the first output corresponding to the encoding layer is determined based on the result of the cross-attention processing.
[0035] Optionally, the diffusion execution unit is used to:
[0036] Acquire a first number, generate a first number of random seeds, and generate a first number of the audio features to be diffused based on the first number of random seeds;
[0037] Inputting the first number of audio features to be diffused into a diffusion model respectively, and performing diffusion processing on the first number of audio features to be diffused based on visual features respectively, so as to obtain a first number of candidate accompanying audios;
[0038] Accompanying audio is determined from among the first number of candidate accompanying audios.
[0039] Optionally, the diffusion adjustment unit is used to:
[0040] Obtaining an initial input to a first encoding layer of the diffusion model;
[0041] Performing zero convolution processing on the semantic features of the target sound event, and superimposing the semantic features of the target sound event after the zero convolution processing and the initial input to obtain the superimposed semantic features of the target sound event;
[0042] The superimposed semantic features of the target sound event are input into a corresponding semantic feature adapter, and a zero convolution process is performed on the output of the semantic feature adapter to obtain the first adjustment feature.
[0043] Optionally, the accompanying audio generating device further includes a second training unit, and the second training unit is used for:
[0044] Copying the coding layer in the diffusion model to obtain the initialized semantic feature adapter, and freezing the parameters of the diffusion model, the semantic feature adapter includes adapter layers corresponding to the plurality of coding layers of the diffusion model respectively;
[0045] Acquire a second training sample set, where the second training sample set includes at least one sample content;
[0046] Extracting sample original audio features, sample target sound event semantic features and sample content features from the sample content;
[0047] Performing multiple noise addition steps on the sample original audio feature through the diffusion model to obtain a noisy audio feature, and storing the forward noise added to the sample original audio feature in each noise addition step;
[0048] The sample content feature is input into a diffusion model so that the diffusion model performs multiple denoising steps on the noise audio feature based on the sample content feature; in each denoising step, the sample target sound event semantic feature is input into a semantic feature adapter to obtain a sample adjustment feature corresponding to each coding layer, and the output of the corresponding coding layer is adjusted based on the sample adjustment feature to obtain the reverse noise corresponding to a single denoising step, and multiple denoising steps correspond to multiple noise addition steps;
[0049] A second loss value is determined based on the reverse noise and the corresponding forward noise of the noise adding step, and the parameters of each adapter layer are optimized based on the second loss value until the semantic feature adapter meets the first condition, thereby obtaining the semantic feature adapter.
[0050] Optionally, the accompanying audio generating device further includes a third training unit, and the third training unit is used to:
[0051] Acquire a third training sample set, wherein the third training sample set includes a plurality of sample audio and video pairs, and the sample audio and video pairs include corresponding training sample audio and training sample video;
[0052] Encoding each of the training sample audios and each of the training sample videos respectively by the audio encoder and the visual encoder to obtain a plurality of training sample audio features and a plurality of training sample visual features;
[0053] The training sample audio and the training sample video in the single sample audio-video pair are respectively divided into a plurality of sample audio segments and a plurality of sample video segments corresponding to each other, and the plurality of sample audio segments and the plurality of sample video segments are respectively encoded by the audio encoder and the visual encoder to obtain a plurality of sample segment audio features and a plurality of sample segment visual features;
[0054] Determine a semantic alignment score based on the multiple training sample audio features and the multiple training sample visual features, and determine a temporal alignment score based on the multiple sample segment audio features and the multiple sample segment visual features;
[0055] An audio-video alignment score is determined based on the semantic alignment score and the temporal alignment score, and the audio encoder and the visual encoder are optimized based on the audio-video alignment score until the audio-video alignment score is greater than an alignment score threshold.
[0056] Optionally, the feature extraction unit is used to:
[0057] Obtaining a timestamp feature of a sound event of the target video;
[0058] The diffusion adjustment unit is used to:
[0059] generating a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature;
[0060] adjusting the second output based on the second adjustment feature to obtain a third output;
[0061] The accompanying audio of the target video is generated based on the third output, so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event according to the timestamp corresponding to the timestamp feature of the sound event.
[0062] Optionally, the feature extraction unit is used to:
[0063] From the target video, identifying a target sound event;
[0064] Obtaining the duration of the target sound event;
[0065] Obtaining the event type of the target sound event;
[0066] If the duration and the event type meet a predetermined condition, a timestamp feature of the sound event of the target video is obtained.
[0067] Optionally, the feature extraction unit is used to:
[0068] If the target sound event is a subject voice event, identifying a subject mouth size in a frame in the target video corresponding to the subject voice event;
[0069] If the size of the object's mouth exceeds a predetermined size, a sound event timestamp feature of the target video is obtained.
[0070] Optionally, the feature extraction unit is used to:
[0071] In response to a first operation of the target terminal on the target video, determining a plurality of first annotation information, wherein the first annotation information is used to mark a time node when a sound event occurs in the target video;
[0072] The sound event timestamp feature is determined based on the plurality of first annotation information.
[0073] Optionally, the feature extraction unit is used to:
[0074] Performing sound event detection based on visual features of the target video to determine at least one sound event in the target video, so as to determine at least one time node corresponding to the at least one sound event in the target video;
[0075] The sound event timestamp feature is determined based on the at least one time node.
[0076] Optionally, the diffusion adjustment unit is used to:
[0077] Acquire a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, wherein a sum of the first weight coefficient and the second weight coefficient is not greater than a first threshold;
[0078] The first adjustment feature and the second adjustment feature are weighted based on the first weight coefficient and the second weight coefficient respectively to obtain a weighted first adjustment feature and a weighted second adjustment feature, and the weighted first adjustment feature and the weighted second adjustment feature are accumulated to the first output of the corresponding coding layer to obtain the third output.
[0079] The disclosed embodiment not only relies on the visual features of the target video to generate the accompanying audio, but also obtains the semantic features of the target sound event of the target video. The semantic features of the target sound event reflect the semantics of the target video. The visual features are input into the coding layer of the diffusion model. Based on the semantic features of the target sound event, a first adjustment feature of the coding layer can be generated. The first output of the coding layer can be adjusted with this first adjustment feature to obtain a second output. Based on the second output, the accompanying audio of the target video can be accurately generated. Since the semantics of each event in the target video is reflected in the first adjustment feature, which is used to adjust the first output of the coding layer, the accompanying audio finally generated can be semantically aligned with the video and will not miss the events in the video.
[0080] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0082] Figure 1 is a structural diagram of a system to which the accompanying audio generation method according to an embodiment of the present disclosure is applied;
[0083] Figure 2A-2E It is a schematic diagram of an application scenario interface diagram of an embodiment of the present disclosure applied in an accompanying audio generation application scenario and a comparison diagram with a traditional method in the related art;
[0084] Figure 3 is a flowchart of a method for generating accompanying audio according to an embodiment of the present disclosure;
[0085] Figure 4 is a schematic diagram of a process of generating accompanying audio based on a target video according to an embodiment of the present disclosure;
[0086] Figure 5 is a schematic diagram of the architecture of a diffusion model according to an embodiment of the present disclosure;
[0087] Figure 6 yes Figure 3 A flowchart of step 320 in FIG.
[0088] Figure 7 yes Figure 3 A flowchart of step 320 in FIG.
[0089] Figure 8 is a schematic diagram of input and output of a coding layer according to an embodiment of the present disclosure;
[0090] Fig. 9 yes Figure 3 A flowchart of step 310 in FIG.
[0091] Fig.10 yes Figure 3 A flowchart of step 310 in FIG.
[0092] Fig.11 is a flow chart of updating the semantic features of a target sound event based on a positive prompt word and a reverse prompt word according to an embodiment of the present disclosure;
[0093] Fig.12 It is a schematic diagram of an interface for obtaining positive prompt words and reverse prompt words according to an embodiment of the present disclosure;
[0094] Fig.13 yes Figure 3 A flowchart of step 330 in FIG.
[0095] Fig.14 It is a flowchart of generating a first adjustment feature based on a target sound event semantic feature and adjusting a first output of a coding layer based on the feature according to an embodiment of the present disclosure;
[0096] Fig.15 is a flow chart of weighting a first adjustment feature based on a weight coefficient and then superimposing it on a first output according to an embodiment of the present disclosure;
[0097] Fig.16 is a schematic diagram of an interface for obtaining a weight coefficient according to an embodiment of the present disclosure;
[0098] Fig.17 is a flow chart of adjusting a diffusion processing process based on a sound event timestamp feature according to an embodiment of the present disclosure;
[0099] Fig.18 is a comparative schematic diagram of a diffusion processing process before and after adjusting the diffusion processing process based on the timestamp feature of a sound event according to an embodiment of the present disclosure;
[0100] Fig.19 yes Fig.17A flowchart of step 1710 in FIG.
[0101] Fig. 20 is a flow chart for determining whether to obtain a timestamp feature of a sound event according to an embodiment of the present disclosure;
[0102] Fig.21 yes Fig.17 A flowchart of step 1710 in FIG.
[0103] Fig. 22 is a schematic diagram of an interface for marking a sound event timestamp according to an embodiment of the present disclosure;
[0104] Fig.23 yes Fig.17 A flowchart of step 1710 in FIG.
[0105] Fig.24 is a schematic diagram of an interface for automatically detecting a sound event timestamp according to an embodiment of the present disclosure;
[0106] Fig.25 yes Fig.17 A flowchart of step 1730 in FIG.
[0107] Fig.26 is a flow chart of training a semantic feature generation model according to an embodiment of the present disclosure;
[0108] Fig. 27 is a flow chart of training a semantic feature adapter according to an embodiment of the present disclosure;
[0109] Fig.28 is a schematic diagram of the structure of a semantic feature adapter according to an embodiment of the present disclosure;
[0110] Fig.29 is a schematic diagram of a process of training a semantic feature adapter according to an embodiment of the present disclosure;
[0111] Fig.30 is a flowchart of training a visual encoder and an audio encoder according to an embodiment of the present disclosure;
[0112] Fig.31 is a specific implementation example process diagram of the accompanying audio generation method of the embodiment of the present disclosure;
[0113] Fig.32 is a block diagram of an accompanying audio generating device according to an embodiment of the present disclosure;
[0114] Fig.33 is a terminal structure diagram according to an embodiment of the present disclosure;
[0115] Fig.34is a server structure diagram according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0116] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.
[0117] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:
[0118] Multimodal model: An artificial intelligence model that can process cross-modal data. It learns how to understand the input information cross-modally by learning the associations between data of different modalities, and then converts and fuses the input multimodal information into a feature representation that the model can understand, and performs the corresponding tasks based on the feature representation to obtain the corresponding output. In related technologies, multimodal models are commonly used in scenarios such as text images, text videos, and multimodal search.
[0119] Vision language model (VLM): A multimodal model for processing visual and text information. It learns the association between image information and text information, performs corresponding tasks based on the input image and text, and generates corresponding text input. It can handle multiple cross-modal data processing tasks such as visual question answering, image captioning, text understanding, and image recognition. It can receive image or text input, understand the semantic information in the image or text input, and generate corresponding text output.
[0120] Diffusion model: A generative model that can be used for cross-modal data generation tasks, such as text-to-image, text-to-video, video-to-speech, etc. The diffusion model is inspired by non-equilibrium thermodynamics. It defines a Markov chain of diffusion steps to slowly add random noise to the data, converting the original data into noisy data, and then learns to reverse the diffusion process to learn how to build a generative model of the required data from the noise. The training process of the diffusion model includes forward diffusion and backward diffusion. During training, the diffusion model first receives the original data of a modality A and the diffusion guide data of another modality corresponding to the original data. The forward diffusion process includes multiple noise addition steps. In each noise addition step, the diffusion model predicts a noise and adds it to the original data of the diffusion model. The original data is transformed into a noisy data composed entirely of noise by gradually adding noise through multiple noise addition steps. At the same time, the diffusion model records the noise added in each noise addition step. After the original input is converted into noise data, multiple denoising steps are performed on the noise data. These denoising steps correspond to the denoising steps one by one. In each denoising step, a noise is predicted based on the diffusion guide data and the noise data of the current denoising step and the predicted noise is subtracted from the noise data. The loss value is calculated by comparing the noise predicted in the denoising step with the noise added to the original data in the corresponding denoising step, so that the diffusion model learns how to predict the corresponding noise based on the data of one modality to generate data of another modality. After the training is completed, when the diffusion model is actually used, the reverse diffusion process of the diffusion model is used to predict the noise corresponding to each denoising step based on the given input, and to denoise the data composed of noise to generate the corresponding output.
[0121] Wensheng Video: A technology that automatically generates videos through artificial intelligence models. It generates the corresponding video by inputting text descriptions into a trained multimodal model, which learns how to capture the semantic information of the text descriptions during the training process and generates a series of continuous video frame images.
[0122] Video is an important form of content expression, and video dubbing is an important part of the video creation process. By dubbing the video, the video picture can be combined with the audio, which can convey a lot of information more intuitively. However, the traditional video dubbing method requires video creators to spend a lot of time and energy. Especially with the development of Vincent video technology, video creators can automatically generate videos through Vincent video's artificial intelligence model. In related technologies, videos generated by artificial intelligence models are often silent videos, and video creators still need to spend a lot of time and energy to dub these videos.
[0123] In the related art, there are some methods for automatically dubbing silent videos through artificial intelligence models. These methods often encode the video into corresponding video features and input the video features into the corresponding audio generation model, so that the audio generation model generates corresponding audio based on the video features. The video features are used to represent the visual information of the video screen. The correspondence between the visual screen and the sound events is relatively complex. It is difficult for the audio generation model to capture all the sound events occurring in the video from the video features that represent the visual information, resulting in the lack of sound effects corresponding to some sound events in the generated audio, making the audio and video generated by the model semantically unaligned.
[0124] Based on this, the present disclosure proposes a method for generating accompanying audio, related devices and media, which can enable the accompanying audio generated based on the target video to include audio clips corresponding to each sound event in the target video, and the generated accompanying audio can be accurately aligned with the target video in semantics.
[0125] System architecture and scenario description of the application of the disclosed embodiment:
[0126] Figure 1 1 is a system architecture diagram of the accompanying audio generation method according to an embodiment of the present disclosure, which includes: an object terminal 110, the Internet 120, a gateway 130, and a server 140.
[0127] The object terminal 110 is a device used by the object to upload the target video so that the server 140 can generate the corresponding audio based on the target video. It includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, car terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and work together using a common display device to form a terminal. The object terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.
[0128] The gateway 130 is also called an internetwork connector or a protocol converter. The gateway 130 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway 130 is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway 130 can also provide filtering and security functions. The message sent by the object terminal 110 to the server 140 is sent to the corresponding server 140 through the gateway 130. The message sent by the server 140 to the object terminal 110 is also sent to the corresponding object terminal 110 through the gateway 130.
[0129] The server 140 refers to a computer system that can provide audio generation services to the target terminal 110. Compared with the target terminal 110, the server 140 has higher requirements in terms of stability, security, performance, etc. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 140 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.
[0130] The embodiments of the present disclosure can be applied in various scenarios, such as FIG. 2A to FIG. 2E The automatic dubbing scene shown.
[0131] Reference Figure 2A In the related art, dubbing for silent videos is often manually performed by dubbing artists using corresponding dubbing tools, which requires dubbing artists to spend a lot of time and energy.
[0132] In this disclosure, reference is made to Figure 2B The subject opens the audio generation software through the terminal. In the interface of the audio generation software, a prompt to upload the video file and an upload box for uploading the video file are displayed. The subject can click the upload box in the software interface and then select the video file for which audio needs to be generated from the local files of the terminal, or drag and drop the video file from the local terminal to the upload box to upload the video file for dubbing to the audio generation software.
[0133] Reference Figure 2C , the subject uploaded a 60-second video file and needs to generate the corresponding accompanying audio. After receiving the video file uploaded by the subject, the audio generation software will display a video file playback window, an option window for setting some control parameters of the accompanying audio method disclosed in the execution process, a preview window for the subsequently generated accompanying audio, and a video preview window obtained after synthesizing the generated accompanying audio into the target video, etc. on the interface of the audio generation software.
[0134] Afterwards, the subject sets some control parameters of the audio generation process in the interface of the audio generation software. Figure 2D As shown, in the parameter setting window of semantic alignment, select automatic generation to make the server automatically call the artificial intelligence model to generate video description text that describes the sound events in the video file. At the same time, the object can also set the weight coefficient of semantic alignment and the batch size. The batch size indicates the number of accompanying audios that need to be generated, such as Figure 2D As shown, the weight coefficient is set to 1 and the batch size is set to 4; after setting the corresponding control parameters in the interface of the audio generation software, such as Figure 2E As shown, the object clicks the option to generate audio, and the audio generation software generates accompanying audio corresponding to the input video file by executing the accompanying audio generation method disclosed in the present invention, and displays the generated 4 candidate accompanying audios in the audio generation result window for the object to select; the object can select these candidate accompanying audios to preview the synthesis result after synthesizing the selected candidate accompanying audio into the video in the synthesized video result window, so as to select the final accompanying audio from the candidate accompanying audios.
[0135] It should be understood that the above content only illustrates some application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied may include but are not limited to the specific embodiments listed above. For example, in the scenario of Vincent video, corresponding accompanying audio can be generated based on the video generated by the Vincent video application, and then the video after synthesizing the accompanying audio is output.
[0136] General description of the embodiments of the present disclosure:
[0137] The accompanying audio generation method of the embodiment of the present disclosure may be executed on the target terminal 110 , may be executed on the server 140 , or may be partially executed on the target terminal 110 and the other partially executed on the server 140 .
[0138] like Figure 3 As shown, according to an embodiment of the present disclosure, a method for generating an accompanying audio is proposed, and the accompanying audio generation method includes:
[0139] Step 310, extracting visual features of the target video and obtaining semantic features of the target sound event of the target video;
[0140] Step 320, inputting the visual features into a diffusion model, the diffusion model including a coding layer for performing diffusion processing on the diffuse audio features according to the visual features;
[0141] Step 330, generates a first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event, adjusts the first output of the coding layer based on the first adjustment feature to obtain a second output, and generates accompanying audio of the target video based on the second output so that the accompanying audio contains an audio clip corresponding to the semantic feature of the target sound event.
[0142] In step 310, the target video refers to the video for which the corresponding accompanying audio needs to be generated. It is understandable that the target video is a silent video. For example, the target video is a model generated by the Vincent video model by inputting a text description into the Vincent video model by the subject, or it can also be a video produced by the subject through other video production tools, such as animations produced by some animation creators.
[0143] Visual features are features used to characterize the visual information of the target video. Figure 4 , visual features can be obtained by encoding the target video through a trained visual encoder. The visual features can be a sequence data or a multi-dimensional tensor. Specifically, in the visual encoder, the target video is first converted into multiple video frames, and then the features of these video frames are extracted through a convolutional neural network, and the features of these video frames are fused through a fully connected layer to obtain the visual features corresponding to the target video. The training process of the visual encoder will be explained later, so I will not go into details here.
[0144] The target sound event semantic feature is a semantic feature used to characterize the target sound event present in the target video. The target sound event refers to an event contained in the video screen of the target video that will produce a corresponding sound effect. For example, in the target video, there is a picture of a dog barking. Then, after detecting the target video, it can be determined that the target sound event "dog barking" exists in the target video. Specifically, the target sound event semantic feature can be obtained by encoding the sound event semantic description, and the sound event semantic description is a text used to describe the sound event in the target video. For example, when the target sound event "dog barking" is detected in the target video, a corresponding sound event semantic description "the sound events present in the video include: dog barking" is generated, and then the sound event semantic description is encoded by the encoder of the text modality to obtain the corresponding target sound event semantic feature. Specifically, the sound event in the target video can be identified by a visual language model, and a corresponding sound event semantic description is generated, and then the sound event semantic description is encoded by the encoder of the text modality to obtain the target sound event semantic feature.
[0145] The details of step 310 will be described in detail later and will not be repeated here.
[0146] In step 320, the diffusion model is a pre-trained model for predicting noise based on visual features to generate corresponding audio.
[0147] The encoding layer is a functional layer composed of some neural network layers in the backbone network of the diffusion model. Figure 5 The diffusion model is based on the Unet network architecture. The diffusion model of the Unet network architecture includes multiple one-to-one corresponding encoders and decoders, and each encoder is a coding layer. In addition, refer to Figure 5 In the diffusion model of the Unet architecture, multiple encoders are cascaded in sequence, and multiple decoders are also cascaded in sequence. An intermediate layer is set between the multiple encoders and the multiple decoders, and this intermediate layer can also be regarded as a coding layer.
[0148] In another embodiment, the diffusion model may also be a model based on a diffusion transformer (DiT) architecture, wherein the diffusion model of the diffusion transformer architecture includes a neural network layer for preprocessing the model input, multiple diffusion transformer blocks (DiT blocks), and a linear layer for decoding the final outputs of the multiple diffusion transformer modules. The multiple diffusion transformer modules are cascaded in sequence, and the output of each diffusion transformer module serves as the input of the next diffusion transformer module, wherein each diffusion transformer module can be regarded as an encoding layer.
[0149] The audio feature to be diffused is the original audio feature before the diffusion processing is performed by the diffusion model. It can be understood that the process of diffusion processing is to perform multiple denoising steps through the diffusion model and predict the corresponding noise in each denoising step, so as to denoise a given data composed of noise, thereby generating a corresponding output. The audio feature to be diffused can be an audio feature composed entirely of random noise, for example, an audio feature composed of Gaussian noise. Specifically, since the audio feature to be diffused is a feature composed of random noise, it is independent of the input of the diffusion model and can be automatically generated by the diffusion model before the diffusion processing is performed, for example, it is generated by random sampling based on the Gaussian distribution by the diffusion model.
[0150] The process of treating diffuse audio features based on visual features is to use visual features as guide vectors to guide the diffusion model to predict the corresponding noise. It can be understood that the diffusion model is a cross-modal generative model. In the process of diffusion processing by the diffusion model, the noise corresponding to each denoising step will be predicted based on the input visual features, and the predicted noise will be subtracted from the initially generated audio features to be diffused, so as to perform multi-step denoising on the diffuse audio features, thereby generating audio data corresponding to the visual features.
[0151] In one embodiment, in the diffusion model of the Unet network architecture, each encoder includes a cross-attention mechanism. Figure 6 In step 320, the diffuse audio feature is diffused according to the visual feature, including:
[0152] Step 610, in the coding layer, determining the intermediate features of the audio to be diffused based on the features of the audio to be diffused;
[0153] Step 620, generating a query matrix based on the intermediate features of the audio to be diffused, and generating a key matrix and a value matrix based on the visual features;
[0154] Step 630, performing cross-attention processing based on the query matrix, the key matrix and the value matrix, and determining the first output corresponding to the encoding layer based on the result of the cross-attention processing.
[0155] In step 610, the intermediate feature of the audio to be diffused refers to the intermediate feature representation of the multi-head attention mechanism input into the coding layer after the audio feature to be diffused is processed accordingly by a part of the neural network layer of the diffusion model. It can be understood that in the diffusion model, each coding layer includes other neural network layers in addition to the multi-head attention mechanism, such as residual blocks, etc. The audio feature to be diffused will first pass through these neural network layers and be processed accordingly when input into the coding layer to obtain the corresponding intermediate feature representation, that is, the intermediate feature of the audio to be diffused, and then input into the multi-head attention mechanism. In addition, multiple coding layers are also cascaded, and the audio feature to be diffused will be processed by each coding layer in turn. Except for the first coding layer, the audio feature to be diffused received by the other coding layers is actually the intermediate feature representation obtained after the previous coding layer performs the corresponding processing on the audio feature to be diffused. Based on this, in an embodiment, in the coding layer, the intermediate feature of the audio to be diffused corresponding to the audio feature to be diffused is first determined, and the intermediate feature of the audio to be diffused is used as the input of the multi-head attention mechanism in the coding layer.
[0156] In step 620, the query matrix is determined based on the intermediate features of the audio to be diffused, and the key matrix and the value matrix are determined based on the visual features. Specifically, in the multi-head attention mechanism, three weight matrices are included, namely the query weight matrix, the key weight matrix and the value weight matrix. These three weight matrices are determined when training the diffusion model and are not described here. The intermediate features of the audio to be diffused are multiplied by the query weight matrix, and the visual features are multiplied by the key weight matrix and the value weight matrix respectively, thereby obtaining the query matrix, the key matrix and the value matrix.
[0157] In step 630, after obtaining the query matrix, key matrix and value matrix, attention calculation is performed based on the query matrix, key matrix and value matrix, so as to inject the information of visual features into the diffusion processing process, so as to guide the diffusion model to predict the corresponding noise based on the visual features. Specifically, the query matrix and the key matrix are first matrix multiplied to obtain the corresponding attention score matrix, and then the attention score matrix is exponentially normalized to obtain the corresponding attention weight matrix, and the value matrix is weighted based on the attention weight matrix to obtain the result of the cross attention processing. After that, the result of the cross attention processing is nonlinearly mapped by the feedforward neural network in the encoding layer, so as to obtain the first output corresponding to the encoding layer.
[0158] In the embodiment disclosed in steps 610 to 630, cross-attention processing is performed based on the audio features to be diffused and the visual features, thereby injecting the information of the visual features into the diffusion processing process, so that the diffusion model diffuses the audio features to be diffused according to the visual features, thereby determining the corresponding first output of the encoding layer based on the visual features.
[0159] In addition, when the diffusion model is a model based on the diffusion converter architecture, the diffusion model includes multiple diffusion converter modules, each diffusion converter module includes a multi-layer perceptron, a multi-head attention mechanism, and a feedforward neural network. After the intermediate data corresponding to the audio feature to be diffused is input into the diffusion converter module, the multi-head attention processing is first performed, and then the output of the multi-head attention mechanism is subsequently processed by the feedforward neural network. The diffusion converter module also includes multiple neural network layers for dimensional scaling and offset processing of the input and output of the multi-head attention mechanism and the feedforward neural network. These neural network layers are connected to the multi-layer perceptron. The diffusion model first encodes the step number corresponding to the visual feature and the denoising step into a corresponding embedding vector through an embedding layer. After that, in each diffusion converter module, a multi-layer perceptron is used to generate multiple dimensional scaling parameters and offset parameters based on the embedding vector, and the input and output of the multi-head attention mechanism and the input and output of the feedforward neural network are dimensional scaled and offset based on these dimensional scaling parameters and offset parameters, so as to inject the information of the visual feature into the diffusion processing process, so that the diffusion model performs diffusion processing on the audio feature to be diffused according to the visual feature.
[0160] Reference Figure 7 In one embodiment, step 320 may further include:
[0161] Step 710, obtaining a first number, generating a first number of random seeds, and generating a first number of audio features to be diffused based on the first number of random seeds;
[0162] Step 720: input the first number of audio features to be diffused into the diffusion model respectively, and perform diffusion processing on the first number of audio features to be diffused based on the visual features to obtain a first number of candidate accompanying audios;
[0163] Step 730: Determine the accompanying audio from the first number of candidate accompanying audios.
[0164] In step 710, the first number represents the number of candidate accompanying audios that the subject wishes to generate based on the input target video. The first number may be the number input by the subject through the terminal. FIG. 2C to FIG. 2E As shown, the object can enter the first number by adjusting the batch size in the interface of the audio generation software of the terminal. In addition, the first number can also be a pre-set number.
[0165] The random seed is the seed for random number generation. Referring to the relevant description of the above step 320, it can be known that the audio feature to be diffused is an audio feature composed entirely of noise, which is randomly sampled based on a given probability distribution and is composed based on the sampled values. The probability distribution is generated based on certain conditions. By using the random seed as the condition for the probability distribution to obey, the probability distribution generated corresponding to different random seeds is also different. By randomly sampling the probability distributions generated based on different random seeds, different audio features to be diffused can be obtained.
[0166] For example, the audio feature to be diffused is an audio feature composed entirely of noise obtained by random sampling based on a Gaussian distribution, and different mathematical expectations and variances will generate different Gaussian distributions. Using the random seed as the mathematical expectation or variance of the Gaussian distribution will result in different Gaussian distributions. Afterwards, by randomly sampling the Gaussian distributions obtained based on different random seeds, different audio features to be diffused can be obtained. For example, by generating four random seeds, namely 1, 2, 3, and 4, four Gaussian distributions can be obtained, which are recorded as G_1, G_2, G_3, and G_4 respectively; by randomly sampling from G_1, an audio feature to be diffused can be obtained; by randomly sampling from G_2, another audio feature to be diffused can be obtained, and so on, the first number of audio features to be diffused is obtained.
[0167] In step 720, after obtaining the first number of audio features to be diffused, the first number of audio features to be diffused are respectively diffused based on the visual features. Since the diffusion model predicts the corresponding noise based on the visual features and the initial audio features to be diffused, and subtracts the predicted noise from the initial audio features to be diffused, thereby generating the corresponding audio corresponding to different audio features to be diffused, the noise predicted by each denoising step in the diffusion processing process is also different, and the initial audio features to be diffused are also different. In this way, after the diffusion processing is completed, the generated audio also has certain differences. In this way, through the diffusion model, the first number of audio features to be diffused are respectively diffused based on the visual features, so that the first number of candidate accompanying audios can be obtained.
[0168] It should be noted that when diffusion processing is performed on each audio feature to be diffused, it is necessary to generate a corresponding first adjustment feature based on the semantic feature of the target sound event, and adjust the first output generated by each encoding layer of the diffusion model during the diffusion processing process to obtain the corresponding second output, and generate the corresponding candidate accompanying audio based on the second output.
[0169] In addition, when diffusion processing is performed on each audio feature to be diffused, the corresponding sound event timestamp feature can also be obtained, and the second adjustment feature corresponding to each coding layer of the diffusion model is generated based on the sound event timestamp feature, and the second output of the coding layer is adjusted to obtain the corresponding third output, and the corresponding candidate accompanying audio is generated based on the third output.
[0170] In step 730, after generating the first number of candidate accompanying audios, the generated first number of candidate accompanying audios can be displayed in the terminal interface of the object. At the same time, in response to the selection operation of the object, the candidate accompanying audios selected by the object are synthesized into the target video, so that the object can preview the result of synthesizing each candidate accompanying audio into the target video in the terminal interface, thereby determining the required accompanying audio from the first number of candidate accompanying audios.
[0171] In the embodiment disclosed in step 710 to step 730, a first number is obtained to generate a first number of different audio features to be diffused, and then the first number of audio features to be diffused are diffused based on the visual features to obtain a first number of candidate accompanying audios. Thus, a plurality of candidate accompanying audios corresponding to the target video are generated at one time, so that the ultimately required accompanying audio can be selected from the first number of candidate accompanying audios, so that the accompanying audio and the target video are better aligned with the target video in terms of semantics and time dimensions.
[0172] In step 330, the first adjustment feature is obtained based on the semantic feature of the target sound event, and is used to inject the information contained in the semantic feature of the target sound event into a feature matrix required for the diffusion processing process. It can be understood that in the process of diffusion processing, the diffusion model will perform corresponding processing on the diffuse audio features based on visual features through multiple coding layers, so as to capture the correlation between the visual features and the output of the diffusion model in different feature dimensions. Accordingly, it is also necessary to inject the information of the semantic feature of the target sound event into the diffusion processing process from different feature dimensions. Therefore, it is necessary to generate the first adjustment feature corresponding to each coding layer based on the semantic feature of the target sound event. Specifically, the first adjustment feature can be obtained by processing the semantic feature of the target sound event through the corresponding semantic feature adapter.
[0173] In one embodiment, the semantic feature adapter can copy the parameters of the structure of the coding layer in the diffusion model, and then train the semantic feature adapter so that the semantic feature adapter learns how to generate the corresponding first adjustment feature based on the semantic feature of the target sound event, so as to inject the information corresponding to the semantic feature of the target sound event into the diffusion processing process. It can be understood that because the initial semantic feature adapter is obtained by copying the parameters and structure of each coding layer of the diffusion model, the structure of the semantic feature adapter is the same as the structure of the multiple coding layers of the diffusion model, and the semantic feature adapter will include multiple adapter layers corresponding to the multiple coding layers of the diffusion model. After the semantic feature of the target sound event is input into the semantic feature adapter, the semantic feature of the target sound event will be processed by each adapter layer in turn, and the corresponding intermediate data will be generated. The intermediate data generated by each adapter layer is the first adjustment feature of the corresponding coding layer. Specifically, the details of using the semantic feature adapter to generate the corresponding first adjustment feature based on the semantic feature of the target sound event and the training process of the semantic feature adapter will be described later, and will not be repeated here.
[0174] The first output refers to the output obtained by each coding layer of the diffusion model after processing the corresponding input. It can be understood that in each denoising step of the diffusion processing process, multiple coding layers of the diffusion model need to perform corresponding processing on the diffuse audio features in turn. After each coding layer performs the corresponding processing task on the diffuse audio features, a corresponding intermediate data will be generated. This intermediate data is the first output corresponding to the coding layer. For example, refer to Figure 8 The input of the coding layer is denoted as x. After the coding layer processes the input x accordingly, the output y of the coding layer is obtained. This output y is used as the input of other coding layers after the coding layer in the diffusion model. This output y is the first output corresponding to the coding layer.
[0175] It should be noted that, refer to Figure 5In the diffusion model of the Unet architecture, the output of each encoding layer is not only used as the input of the corresponding decoding layer, but also as the input of the next encoding layer. At this time, the first adjustment feature is only used to adjust the output from the encoding layer to the corresponding decoding layer, but not the output from the encoding layer to the next encoding layer. For example, refer to Figure 5 In the encoding layer 1, after the diffuse audio features are processed accordingly based on the visual features, the obtained output will be input to the decoding layer 1 and the encoding layer 2 at the same time. At this time, based on the corresponding first adjustment features, the data output from the encoding layer to the decoding layer 1 is adjusted, but there is no need to adjust the output from the encoding layer 1 to the encoding layer 2.
[0176] Specifically, in one embodiment, the first adjustment feature and the first output of the corresponding encoding layer may be superimposed to adjust the first output based on the first adjustment feature, thereby injecting information of the semantic features of the target sound event into the diffusion processing process, so that the accompanying audio finally generated by the diffusion model can contain audio clips corresponding to all sound events represented by the semantic features of the target sound event.
[0177] In the embodiment disclosed in steps 310 to 330, by obtaining the semantic features of the target sound event corresponding to the target video during the diffusion processing, generating the first adjustment features corresponding to each coding layer of the diffusion model based on the semantic features of the target sound event, and adjusting the first output generated by each coding layer of the diffusion model during the diffusion processing of the diffused speech features, and fine-tuning the intermediate data of the diffusion processing process, the information reflected by the semantic features of the target sound event is injected into the diffusion processing process, thereby ensuring that the diffusion model can capture all sound events in the target video during the diffusion processing. As a result, the accompanying audio finally generated by the diffusion model can contain audio clips corresponding to all target sound events in the target video, so that the generated accompanying audio and the target video can be accurately aligned in semantics.
[0178] The above describes the steps 310 to 330 in their entirety. Since the step 320 has been described in detail above, only the details of the steps 310 and 330 will be described in detail below.
[0179] Detailed description of step 310:
[0180] In step 310, the visual features of the target video are extracted, and the semantic features of the target sound event of the target video are obtained;
[0181] In one embodiment, the sound events identified from the target video may include some sound events corresponding to noise or ambient sound. For example, in a rainy scene, the sound event of rain will be identified, and the generated accompanying audio will include the sound of rain. However, for the target video, the sound of rain may not be necessary, and the sound of rain may even affect the sound effects corresponding to other sound events in the target video. In this case, it is necessary to filter out these sound events. Based on this, in one embodiment, referring to Fig. 9 In step 310, the semantic features of the target sound event of the target video are obtained, including:
[0182] Step 910, identifying a sound event from a target video, and acquiring a target frame containing the sound event;
[0183] Step 920, inputting the target frame and the context frame of the target frame in the target video into a sound event importance prediction model to obtain the predicted importance of the sound event;
[0184] Step 930, obtaining the sound intensity corresponding to the sound event;
[0185] Step 940, determining a target sound event from among the identified sound events based on the importance and the sound intensity;
[0186] Step 950, obtaining the target sound event semantic features of the target sound event.
[0187] In step 910, a sound event refers to an event that may exist in the target video and will produce a corresponding sound effect, and it may include all sound events identified from the target video. After identifying these sound events, the video frame corresponding to each sound event is obtained from the target video as the target frame. For example, in the target video, the 5th second of the target video is a rainy scene. Based on the video scene, it is identified that the target video contains the sound event of rain. At this time, the video frame at the 5th second of the target video is used as the target frame corresponding to this sound event. It should be noted that a sound event may correspond to multiple video frames. At this time, the target frame may be the first video frame where the sound event occurs.
[0188] In step 920, the importance prediction model can be a pre-trained model for predicting the importance of sound events, and the importance represents the relevance of the sound event to the content of the target video. The context frame can be a plurality of video frames in a time interval centered on the target frame. For example, the target frame is the video frame corresponding to the 5th second in the target video, and the context frame can be a plurality of video frames in the time period from one second before the target frame to one second after the target frame, that is, a plurality of video frames in the time period from the 4th second to the 6th second of the target video. Exemplarily, a corresponding time interval can be determined based on the position of the target frame in the target video, and then uniform sampling or random sampling is performed within this time interval to obtain the corresponding context frame.
[0189] After obtaining the context frame, the target frame and the context frame are input into the importance prediction model together for importance prediction, so as to determine the sound events with lower importance in the target video. It is understandable that the target video often contains multiple sound events, and the importance of different sound events is often different. For example, the target video is a person playing a musical instrument indoors. During the video perspective switching process, there are a few video frames that show rain outside from the window. Then, from the target video, the sound event of the person playing the musical instrument and the sound event of rain will be identified. However, the correlation between the sound event of rain and the content of the target video is not high. If the sound effect (i.e., the sound of rain) corresponding to the sound event of rain is added to the accompanying audio, it will become noise, affecting the sound effect of the person playing the musical instrument. At this time, the target frame and context of the sound event of rain are input into the importance prediction model, and the importance prediction model predicts that the importance of the sound event is low.
[0190] In step 930, the sound intensity may refer to the loudness corresponding to the sound event. It is understandable that in the target video, some sound events are not very important, but they are sound events with high sound intensity. If the generated accompanying audio does not contain the sound effects of these sound events, it may cause poor viewing experience for viewers. For example, the target video is a person playing a musical instrument indoors. During the video's perspective switching process, there are a few video frames that show thunder and lightning outside the window. Then, from the target video, the sound event of the person playing the musical instrument and the sound event of thunder and lightning will be identified. Although the sound event of thunder and lightning is not highly correlated with the content of the target video and does not last long, if the target video does not have the corresponding sound effect when playing the corresponding thunder and lightning picture, it will also cause the viewer of the target video to have a poor viewing experience. Based on this, in addition to predicting the importance of each sound event in the target video, it is also necessary to determine the sound intensity of each sound event to determine the target sound event.
[0191] Specifically, various sound events can be classified in advance, and the sound intensity interval corresponding to each category can be set. For example, sound events of rigid object collisions such as hammering often have high sound intensity and can be used as a category; sound events of animal calls can be used as a category, etc. After the corresponding sound event is identified from the target video, the category to which the identified sound event belongs is determined, and based on the sound intensity interval corresponding to the category, the sound intensity corresponding to the sound event is determined.
[0192] In step 940, specifically, a first score can be determined according to the importance of the sound event, and the higher the importance of the sound event, the higher the corresponding first score; a second score can be determined according to the sound intensity corresponding to the sound event, and the higher the sound intensity of the sound event, the higher the corresponding second score; the first score and the second score are weighted and summed based on a preset coefficient to obtain a total score corresponding to the sound event, and when the total score of the sound event is greater than a given score threshold, the sound event is determined to be a target sound event. In this way, the identified sound events with low importance and low corresponding sound intensity, that is, some sound events that may correspond to the background sound of the target video, are filtered out to prevent these unimportant background sounds from affecting the sound effects of other sound events in the target video, thereby improving the quality of the generated accompanying audio.
[0193] In step 950, after the target sound event is determined, a text for describing the target sound event, namely, a sound event semantic description, is generated, and the sound event semantic description is encoded through a semantic feature generation model to obtain corresponding target sound event semantic features.
[0194] In step 910 to step 950, after the sound events in the target video are identified, the importance of each sound event is predicted, and the sound intensity corresponding to the sound event is determined; based on the importance and sound intensity of the sound event, the target sound event is determined from all the identified sound events, so that the sound events with low importance and low sound intensity in the target video are filtered out, and then the target sound event semantic features are generated based on the target sound event. In this way, when the accompanying audio is subsequently generated based on the semantic features of the target sound event, unnecessary noise in the accompanying audio can be reduced, and the quality of the generated accompanying audio can be improved.
[0195] In one embodiment, referring to Fig.10 Step 310 may further include:
[0196] Step 1010, decomposing the target video into video frames;
[0197] Step 1020, inputting the decomposed video frames and preset prompts into a visual language model, so that the visual language model generates a semantic description of the sound event, wherein the preset prompts are used to prompt the visual language model to generate a semantic description of the sound event based on the video frames;
[0198] Step 1030, encode the semantic description of the sound event through a semantic feature generation model to obtain the semantic features of the target sound event.
[0199] In step 1010, a video frame refers to a single frame image in a target video. Specifically, all video frames of the target video can be directly extracted; or, the target video can be decomposed into multiple segments, and sampling is performed from each segment to obtain multiple video frames corresponding to the target video; or, the target video can be uniformly sampled based on a given time interval to extract corresponding video frames from the target video; in addition, the difference between two adjacent video frames in the target video can be calculated to determine whether there is a large change between the two adjacent video frames, thereby extracting relatively critical video frames in the target video, and thus decomposing the target video into video frames.
[0200] In step 1020, the visual language model is a multimodal model that can capture information in the image modality and information in the text modality and perform corresponding processing to generate corresponding text output. For example, the visual language model can be a multimodal large language model that supports input images in the related technology; or it can be a multimodal model dedicated to processing visual language tasks, such as the Mini-Gemini model. The visual language model can accurately capture the association between multiple input image information, and thereby identify events that occur in multiple input images. The preset prompt is a prompt used to prompt the visual language model to detect sound events contained in the input video frame based on the input video frame. For example, the preset prompt can be "Please identify the sound events contained in this set of images." Refer to Figure 4 , the extracted video frames and preset prompts are input into the visual language model together, so that the visual language model recognizes the sound events in the input video frames, thereby generating corresponding semantic descriptions of the sound events. In this way, by utilizing the ability of the visual language model to process image information, the sound events contained in the target video are identified from the video frames. It can be understood that the semantic description of the sound event is a text used to describe the sound events contained in the target video.
[0201] In step 1030, the semantic feature generation model is a text modal encoder, and the text information is encoded. It should be noted that the semantic feature generation model can encode the text information into text features that have a high degree of similarity with the video features of a video with the same semantic information. For example, a text is "a puppy is barking", and the corresponding video is a video of "a puppy is barking". Then the text features obtained by encoding the text through the semantic feature generation model are highly similar to the video features obtained by encoding the video of "a puppy is barking". After obtaining the semantic description of the sound event, refer to Figure 4 , the semantic description of the sound event is encoded through the semantic feature generation model to obtain the corresponding semantic features of the target sound event. Specifically, the training process of the semantic feature generation model will be described later and will not be repeated here.
[0202] In the embodiment disclosed in steps 1010 to 1030, the target video is decomposed into multiple video frames, and the video frames and preset prompts are input into the visual language model. The ability of the visual language model to process images is utilized to identify sound events in the target video based on the video frames to generate corresponding semantic descriptions of the sound events, and thereby generate corresponding semantic features of the target sound events.
[0203] In one embodiment, referring to Fig.11 , after step 1030, further comprising:
[0204] Step 1110, obtaining a positive prompt word and a reverse prompt word, and encoding the positive prompt word and the reverse prompt word into corresponding prompt word features; wherein the positive prompt word is used to prompt the diffusion model to correspond to a first reference sound event in the generated accompanying audio, and the reverse prompt word is used to prompt the diffusion model to correspond to a second reference sound event in the generated accompanying audio;
[0205] Step 1120, concatenating the prompt word feature to the target sound event semantic feature to update the target sound event semantic feature.
[0206] In step 1110, the positive prompt word is a prompt word corresponding to the first reference sound event that the diffusion model needs to add to the accompanying audio, and the first reference sound event is a sound event that the subject hopes to have a corresponding audio clip in the generated accompanying audio; the reverse prompt word is a prompt word corresponding to the second reference sound event that the diffusion model cannot correspond to in the generated accompanying audio, and the second reference sound event is a sound event that does not have a corresponding sound effect in the generated accompanying audio. For example, if the target video is a video of a pet dog playing, then the positive prompt word can be: dog barking; the reverse prompt word can be a pre-set prompt word for indicating a sound event corresponding to a sound that is irrelevant to most target videos, for example, the reverse prompt word can be: noise, background music, etc. In addition, the reverse prompt word can also be used to indicate a sound that may be related to the target video, but the subject does not want to appear in the accompanying audio in most scenes, such as: a sharp and harsh sound, etc.
[0207] The forward prompt word and the reverse prompt word can also be prompt words input by the subject through the terminal. Fig.12 The subject can select the option of inputting prompt words in the interface of the audio generation software, and then a text input box for inputting positive prompt words and reverse prompt words will be displayed in the interface of the audio generation software, and the subject can input the corresponding positive prompt words and reverse prompt words in the corresponding text input boxes.
[0208] After obtaining the forward prompt word and the reverse prompt word, which are also information in the text mode, based on this, similar to the semantic description of the sound event, the forward prompt word and the reverse prompt word can be encoded into corresponding prompt word features through the semantic feature generation model. It should be noted that since the reverse prompt word is used to prompt the sound event that cannot be corresponding in the accompanying audio, when encoding the reverse prompt word into the prompt word feature, the reverse prompt word and the corresponding reverse prompt template need to be encoded together. For example, the reverse prompt template can be "The following sound events do not need to be included:". This avoids misidentifying the reverse prompt word as a forward prompt word when the prompt word feature information is injected into the diffusion processing process, resulting in the final generated accompanying audio carrying an audio clip of the sound event corresponding to the reverse prompt word.
[0209] After obtaining the prompt word feature, since both the prompt word feature and the target sound event semantic feature are used to describe the sound event corresponding to the accompanying audio, based on this, the prompt word feature can be spliced to the target sound event semantic feature, thereby updating the target sound event semantic feature. Subsequently, the corresponding first adjustment feature is generated based on the updated target sound event semantic feature, and the information carried by the positive prompt word and the reverse prompt word can be injected into the diffusion processing process, so that the accompanying audio finally generated by the diffusion model can carry the audio clip of the sound event corresponding to the positive prompt word, and does not carry the audio clip of the sound event corresponding to the reverse prompt word.
[0210] In step 1110 to step 1120, by obtaining positive prompt words and reverse prompt words and generating corresponding prompt word features, and then splicing the prompt word features to the target sound event semantic features, the object can manually input positive prompt words and reverse prompt words, thereby selectively injecting the semantic information of some sound events into the diffusion processing process, and at the same time avoid the situation where the visual language model fails to accurately identify all sound events in the target video or mistakenly identifies sound events that are not included in the target video.
[0211] Detailed description of step 330:
[0212] In step 330, a first adjustment feature corresponding to the coding layer is generated based on the target sound event semantic feature, and a first output of the coding layer is adjusted based on the first adjustment feature to obtain a second output, and an accompanying audio of the target video is generated based on the second output.
[0213] Reference Fig.13 In step 330, generating a first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event includes:
[0214] Step 1310, obtaining an initial input of a first coding layer of a diffusion model;
[0215] Step 1320, performing zero convolution processing on the semantic features of the target sound event, and superimposing the semantic features of the target sound event after the zero convolution processing and the initial input to obtain the superimposed semantic features of the target sound event;
[0216] Step 1330, input the superimposed semantic features of the target sound event into the corresponding semantic feature adapter, and perform zero convolution processing on the output of the semantic feature adapter to obtain the first adjusted feature, wherein the semantic feature adapter is obtained by training after freezing the parameters of the diffusion model.
[0217] In step 1310, since the semantic feature adapter needs to determine the first adjustment features corresponding to each coding layer of the diffusion model based on the corresponding input to adjust the first output of each coding layer, when determining the first adjustment features, it is necessary to consider not only the input semantic features of the target sound event but also the inputs of the multiple coding layers of the diffusion model. Based on this, the initial input of the first coding layer among the multiple coding layers of the diffusion model is obtained, that is, the audio feature to be diffused.
[0218] In step 1320, refer to Fig.14 A zero convolution layer is set at the input end of the semantic feature adapter. The zero convolution layer is a 1*1 convolution kernel, and its weight and bias are determined in the process of training the semantic feature adapter. Before the target sound event semantic feature or sound event timestamp feature is input to the corresponding semantic feature adapter, it will first be processed by the corresponding zero convolution layer, and then superimposed with the initial input of the first encoding layer obtained from the diffusion model through an adder to obtain the superimposed target sound event semantic feature as the input of the semantic feature adapter.
[0219] In step 1330, the semantic features of the superimposed target sound event are input into the corresponding semantic feature adapter. The semantic feature adapter is obtained by copying the structure and parameters of the coding layer of the diffusion model, then freezing the parameters of the diffusion model for training, and optimizing the parameters of the semantic feature adapter. The semantic feature adapter has the same network structure as the coding layer of the diffusion model, and the semantic feature adapter includes adapter layers corresponding to the various coding layers of the diffusion model. After the semantic features of the superimposed target sound event are input into the semantic feature adapter, the semantic features of the superimposed target sound event are processed in turn by the various neural network layers in the semantic feature adapter, and the output corresponding to each adapter layer can be obtained. Specifically, since the semantic feature adapter has the same neural network structure as the coding layer of the diffusion model, the difference is that the parameters of each layer are different. Based on this, the processing process performed by the semantic feature adapter on the semantic features of the superimposed target sound event is the same as the processing process performed by the various coding layers of the diffusion model on the diffuse audio features, and the difference is that the parameters in the processing process are the parameters of the corresponding neural network layers. Based on this, the processing process performed by the semantic feature adapter on the semantic features of the superimposed target sound event can refer to the processing process performed by each encoding layer of the diffusion model on the diffuse audio features when the diffuse audio features are diffusely processed according to the visual features in the above step 320. For example, when the diffusion model is a model based on the Unet architecture, each adapter layer of the semantic feature adapter also includes a cross-attention mechanism and other neural network layers. After the semantic features of the superimposed target sound event are input into the semantic feature adapter, the corresponding processing is performed through the neural network layer in the adapter layer, and in the cross-attention mechanism, cross-attention processing is performed based on the visual features and the semantic features of the superimposed target sound event.
[0220] After obtaining the output of each adapter layer, since each adapter layer of the semantic feature adapter is connected to the output of the corresponding encoding layer in the diffusion model through a zero convolution layer, based on this, it is necessary to perform zero convolution processing on the output of each adapter layer, so as to obtain the first adjustment features corresponding to each encoding layer. Similar to step 1320, the weights and biases of these zero convolution layers are also determined in the process of training the semantic feature adapter, which will not be repeated here. It should be noted that since the corresponding zero convolution processing is performed through different zero convolution layers in steps 1320 and 1330, the parameters of different zero convolution layers are different, that is, in steps 1320 and 1330, the weights and bias parameters when performing zero convolution processing are different.
[0221] Exemplarily, steps 1310 to 1330 may be expressed as:
[0222] ;
[0223] in represents the initial input, i.e., the audio feature to be diffused; Represents the semantic features of the target sound event; represents the first adjustment feature; Represents the parameters of each adapter layer of the semantic feature adapter, Indicates that each adapter layer is based on the corresponding parameters Perform appropriate processing on the input; represents the parameters of the zero convolutional layer at the input of the semantic feature adapter, Indicates that based on parameters Perform zero convolution processing on the input semantic features of the target sound event; represents the parameters of the zero convolutional layer connected to the output of the adapter layer of the semantic feature adapter, Indicates that based on parameters Zero convolution is performed on the output of the adapter layer.
[0224] In the embodiment disclosed in steps 1310 to 1330, zero convolution processing is performed on the input semantic features of the target sound event, and the semantic features of the target sound event after zero convolution processing are superimposed with the initial inputs of multiple coding layers of the diffusion model to obtain the superimposed semantic features of the target sound event. After that, the superimposed semantic features of the target sound event are input into the trained semantic feature adapter, so that the superimposed semantic features of the target sound event are processed accordingly through each adapter layer of the semantic feature adapter to obtain the output of each adapter layer, and then the output of each adapter layer is subjected to zero convolution processing. Thus, the first adjustment features corresponding to each coding layer are generated according to the semantic features of the target sound event through the semantic feature adapter.
[0225] The above steps 1310 to 1330 show the process of generating the first adjustment feature based on the semantic feature of the target sound event. The subsequent process of generating the second adjustment feature based on the timestamp feature of the sound event can refer to the detailed process of steps 1310 to 1330, that is, the sound event timestamp feature after zero convolution processing is superimposed on the initial input and input into the timestamp feature adapter to perform corresponding processing, and then the output of each adapter layer is processed by zero convolution through the corresponding zero convolution layer to obtain the second adjustment feature corresponding to each coding layer, which will not be repeated here.
[0226] In one embodiment, referring to Fig.15 , the accompanying audio generation method also includes:
[0227] Step 1510, obtaining a weight coefficient corresponding to the first adjustment feature;
[0228] Step 1520: weight the first adjustment feature based on the weight coefficient to obtain a weighted adjustment feature, and superimpose the weighted adjustment feature with the output of the corresponding coding layer.
[0229] In step 1510, the weight coefficient is used to adjust the diffusion processing process based on the semantic features of the target sound event. In one embodiment, the weight coefficient can be input by the object through the terminal, for example, referring to Fig.16 A pull bar for controlling the weight coefficient is provided in the interface of the audio generation software, and the object forms the input weight coefficient by adjusting the pull bar in the interface of the audio generation software.
[0230] In step 1520, after obtaining the corresponding weight coefficient, the first adjustment feature is weighted based on the weight coefficient. Specifically, the first adjustment feature is a matrix with the same shape as the output of the coding layer, and the weight coefficient is an input constant. By multiplying the weight coefficient and each element in the first adjustment feature, the first adjustment feature is weighted based on the weight coefficient to obtain the weighted adjustment feature. The weighted adjustment feature is then superimposed with the first output of the corresponding coding layer, so as to adjust the first output of the coding layer based on the weighted first adjustment feature. Specifically, steps 1510 to 1520 can be expressed as: ,in represents the first output of the encoding layer, represents the weight coefficient, represents the first adjustment feature, Represents the second output based on the encoding layer.
[0231] In the embodiment disclosed in steps 1510 to 1520, the first adjustment feature is weighted by obtaining the weight coefficient corresponding to the first adjustment feature, and then the weighted first adjustment feature is superimposed on the first output of the corresponding encoding layer. In this way, the object can control the adjustment amplitude of the diffusion processing process based on the semantic characteristics of the target sound event by adjusting the size of the weight coefficient, so as to better control the accompanying audio generation process according to the needs of the object.
[0232] Implementation of a process for adjusting diffusion processing based on sound event timestamp characteristics:
[0233] In one embodiment, the target video may contain some short sound events, such as the sound events of two rigid objects colliding with each other, such as hammering. When generating the accompanying audio, for these short sound events, if the sound effects corresponding to these sound events are not accurately aligned with the corresponding video images in terms of timestamps, it will cause obvious asynchrony between the audio and the image. For example, there is a hammering image in the target video, and in the generated accompanying audio, the hammering sound effect occurs 0.5 seconds before the hammering image. In this case, the quality of the accompanying audio is relatively low. Based on this, in one embodiment, referring to Fig.17 , the accompanying audio generation method also includes:
[0234] Step 1710, obtaining the timestamp feature of the sound event of the target video;
[0235] Step 1720, generating a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature, and adjusting the second output based on the second adjustment feature to obtain a third output;
[0236] Step 1730, generating accompanying audio for the target video based on the third output, so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event according to the timestamp corresponding to the timestamp feature of the sound event.
[0237] In step 1710, the sound event timestamp feature is a feature used to characterize the time node when the sound event in the target video occurs. Based on the sound event timestamp feature, the time node at the beginning of each sound event in the target video can be determined. For example, a hammering sound event occurs at the 5th second of the target video, and the 5th second of the target video is the timestamp corresponding to the hammering sound event. After identifying the timestamp of each sound event in the target video, these timestamps are encoded to obtain the corresponding sound event timestamp feature. The details of step 1710 will be described later and will not be repeated here.
[0238] In step 1720, after obtaining the sound event timestamp feature, the sound event timestamp feature is input into a trained timestamp feature adapter, and the sound event timestamp feature is processed accordingly by the timestamp feature adapter to determine the second adjustment feature corresponding to each coding layer. Among them, the timestamp feature adapter is obtained by freezing the parameters of the diffusion model and then training. The trained timestamp feature adapter can learn how to accurately process the input sound event timestamp feature into a second adjustment feature that can inject the information contained in the sound event timestamp feature into the diffusion processing process. Based on this, after obtaining the second adjustment feature, the output of the coding layer of the diffusion model is adjusted based on the second adjustment feature, so that the accompanying audio generated by the diffusion model can be accurately aligned with the target video in the time dimension. The training process of the timestamp feature adapter will be described later and will not be repeated here.
[0239] Specifically, the structure of the timestamp feature adapter is consistent with the structure of the semantic feature adapter. The details of generating the second adjustment feature corresponding to the coding layer based on the sound event timestamp feature by the timestamp feature adapter can be referred to the above steps 1310 to 1330, and the process of determining the first adjustment features corresponding to each coding layer according to the semantic features of the target sound event will not be repeated here.
[0240] After obtaining the second adjustment feature corresponding to each coding layer, the timestamp information contained in the sound event timestamp feature is injected into the diffusion processing process by superimposing the second adjustment feature to the second output corresponding to the coding layer, so that the accompanying audio finally generated by the diffusion model can be accurately aligned with the target video in the time dimension. For example, in the diffusion model of the Unet architecture, each coding layer will pass the corresponding first output to the corresponding decoding layer for decoding, so as to predict the noise corresponding to the current denoising step. After adjusting the first output of the coding layer to the second output through the first adjustment feature, the second adjustment feature can be superimposed on the second output passed from the coding layer to the decoding layer to obtain the third output, and the third output is used as the input of the decoding layer corresponding to the coding layer.
[0241] In step 1730, after adjusting the second output of the encoding layer to obtain the third output, the neural network layer in the diffusion model for receiving the output of the encoding layer will perform subsequent processing based on the third output, thereby predicting the corresponding noise, and gradually subtracting the noise predicted by each denoising step from the audio feature to be diffused to generate the accompanying audio. For example, in the diffusion model of the Unet architecture, the third output will be used as the input of the decoding layer corresponding to the encoding layer, and the decoding layer will decode based on the input third output, thereby predicting the noise of the corresponding denoising step, and denoising the audio to be diffused to generate the accompanying audio.
[0242] In the embodiment disclosed in steps 1710 to 1730, by obtaining the sound event timestamp feature corresponding to the target video, determining the second adjustment feature corresponding to each coding layer based on the sound event timestamp feature, adjusting the second output corresponding to each coding layer based on the second adjustment feature to obtain a third output, thereby adjusting the intermediate data of the diffusion model during the diffusion processing process, injecting the timestamp information contained in the sound event timestamp feature into the diffusion processing process of the diffusion model, and allowing the diffusion model to perform subsequent processing based on the third output, thereby the finally generated accompanying audio can be more accurately aligned with the target video in the time dimension.
[0243] For example, refer to Fig.18 , Fig.18 The left side in the middle is the audio generation result when the diffusion processing process is not adjusted by the sound event timestamp feature, and the right side is the audio generation result when the diffusion processing process is adjusted by the sound event timestamp feature. It can be seen that in scenes such as shooting, hammering, and sharpening, after adjusting the diffusion processing process by the sound event timestamp feature, the time nodes of the sound events and the corresponding video images can be better aligned.
[0244] In one embodiment, referring to Fig.19 , step 1710, comprising:
[0245] Step 1910, identifying a target sound event from a target video;
[0246] Step 1920, obtaining the duration of the target sound event;
[0247] Step 1930, obtaining the event type of the target sound event;
[0248] Step 1940: If the duration and event type meet the predetermined conditions, obtain the sound event timestamp feature of the target video.
[0249] In step 1910, an event detection model may be used to perform event detection on the target video, and then the detected events may be classified, so that the events that will generate corresponding sounds among the detected events may be determined as target sound events.
[0250] In step 1920, the duration of the target sound event refers to the duration of the target sound event in the target video. Specifically, the duration of the target sound event can be determined by detecting the timestamps of the start frame and the end frame corresponding to the target sound event in the target video. For example, if the timestamp corresponding to the start frame of a sound event in the target video is the 5th second of the target video, and the timestamp corresponding to the end frame in the target video is the 5.5th second of the target video, then the duration of the sound event is 0.5 seconds.
[0251] In step 1930, the event type is used to characterize the classification result to which the target sound event belongs. Specifically, a classification table can be established in advance, and the classification table can be pre-constructed based on expert experience, and the event type of the detected sound event is determined by comparing the classification table. The event type may include: a first event type, a second event type, and a third event type. Specifically, the first event type refers to a sound event type that needs to be accurately aligned. The first event type may be a sound event with a short duration and a relatively prominent corresponding sound effect (for example, a high pitch or a louder loudness), such as a sound event such as a thunder strike; the second event type refers to a sound event type that needs to be combined with the duration to determine whether alignment is required. The sound event of the second event type does not need to be accurately aligned when the duration is long, but needs to be accurately aligned when the duration is short. For example, the sound event of hammering, if it is a single or a few hammerings, then accurate alignment is required, but for long-term continuous hammering, accurate alignment may not be required; the third event type is a sound event type that does not require accurate alignment, which refers to a sound event that often lasts for a long time and has a non-prominent sound effect, such as a sound event such as rain.
[0252] In step 1940, the predetermined condition is a condition for determining whether the sound effect corresponding to the sound event needs to be accurately aligned with the target video. When there is a target sound event corresponding to the first event type in the target video, it is determined that the predetermined condition is met, and the sound event timestamp feature of the target video is obtained; when there is a target sound event corresponding to the second sound event type in the target video, and the duration corresponding to the target sound event is less than a given duration threshold, it is determined that the predetermined condition is met, and the sound event timestamp feature of the target video is obtained. The steps for obtaining the sound event timestamp feature of the target video will be described later and will not be repeated here.
[0253] In the embodiment disclosed in steps 1910 to 1940, the target sound event in the target video is identified, and then the duration and event type of the target sound event are detected. Then, it is determined whether the target video contains a sound event that needs to be accurately aligned according to a predetermined condition. When the duration and event type meet the predetermined condition, it is determined that the sound event timestamp feature corresponding to the target video needs to be obtained, so as to subsequently generate the corresponding second adjustment feature based on the sound event timestamp feature, thereby injecting the information contained in the sound event timestamp feature into the diffusion processing process, ensuring that the generated accompanying audio can be accurately aligned with the target video in the time dimension.
[0254] In one embodiment, the target video may contain objects, which may be anthropomorphic animals or people. In the target video, the objects may talk to each other, or make corresponding sounds through their mouths, such as laughing or shouting. These sounds and the mouth movements of the objects need to be accurately aligned in time. At this time, it is necessary to obtain the corresponding sound event timestamp features. Based on this, in one embodiment, referring to Fig. 20 , the accompanying audio generation method also includes:
[0255] Step 2010, if the target sound event is an object voice event, identifying the object mouth size in the frame corresponding to the object voice event in the target video;
[0256] Step 220: If the size of the mouth of the object exceeds a predetermined size, obtain the sound event timestamp feature of the target video.
[0257] In step 210, the object voice event refers to a sound event in which the object makes a sound through the mouth, for example, the object makes a sound event such as laughing, crying or even talking through the mouth. For example, if the target video contains a picture of a child laughing, this picture corresponds to an object voice event. When the object voice event is detected in the target video, the size of the mouth of the corresponding object is detected to determine whether the mouth picture of the object can be relatively clearly displayed in the target video.
[0258] In step 2020, it can be understood that if the size of the mouth of the object is not larger than the predetermined size, that is, the mouth of the object is actually relatively inconspicuous in the picture of the target video, and the object may even be facing away in the video picture, even if the corresponding sound from the mouth is not aligned with the video picture through the timestamp feature, it will not cause obvious asynchrony between the sound and the picture. When the mouth size is larger than the predetermined size, this indicates that the mouth of the object can be clearly seen from the picture of the target video. At this time, it is necessary to accurately align the corresponding sound effects and mouth movements in the time dimension. Based on this, when the mouth size of the object exceeds the predetermined size, the timestamp feature of the sound event in the target video is obtained to ensure that the generated accompanying audio can be accurately aligned with the target video in the time dimension, avoid obvious asynchrony between the sound and the picture, and improve the quality of the generated accompanying audio.
[0259] In step 2010 to step 2020, by detecting whether there is an object voice event in the target video, when there is an object voice event, detecting the size of the object's mouth, and when the object's mouth size is larger than a predetermined size, obtaining the sound event timestamp feature of the target video, so as to subsequently generate a corresponding second adjustment feature based on the sound event timestamp feature and adjust the second output of the encoding layer, thereby injecting the information of the sound event timestamp feature into the diffusion processing process, ensuring that the generated accompanying audio can also be accurately aligned with the target video in the time dimension, thereby improving the quality of the generated accompanying audio.
[0260] In one embodiment, referring to Fig.21 , step 1710, comprising:
[0261] Step 2110, in response to a first operation of the target terminal on the target video, determining a plurality of first annotation information, wherein the first annotation information is used to mark a time node at which a sound event occurs in the target video;
[0262] Step 2120: Determine a timestamp feature of the sound event based on the plurality of first annotation information.
[0263] In step 2110, the first operation is an operation of the object terminal marking in the target video, and the first marking information is used to represent the time node when the object marks in the target video through the terminal.
[0264] Specifically, refer to Fig. 22 ,In the interface of the audio generation software, the subject can select the option of marking the ,timestamp, and then in the interface of the audio generation software, the target video will be played and the time node of the ,currently played frame in the target video will be displayed, for example, Fig. 22 In the audio generation software interface, the current playback reaches the 5th second of the target video. The audio generation software interface also includes an option for marking the timestamp, i.e., the option to add a timestamp. When the subject clicks on the option in the audio generation software interface, a corresponding first annotation information will be generated to mark the time node of the current playback of the target video. For example, when the playback reaches the 5th second of the target video, the subject clicks on the option to add a timestamp in the audio generation software interface, and a first annotation information corresponding to the 5th second of the target video will be generated.
[0265] In step 2120, each first annotation information is actually a timestamp information. Multiple first annotation information are concatenated and then encoded to obtain the corresponding sound event timestamp feature.
[0266] In the embodiment disclosed in step 2110 to step 2120, the time nodes of multiple sound events in the target video are determined by obtaining the first annotation information input by the object, and the corresponding sound event timestamp features are generated thereby, so as to selectively annotate the time nodes of the sound events in the target video, thereby preventing the model from being unable to identify the timestamps corresponding to all sound events in the target video. Then, the sound event timestamp features are generated based on the first annotation information, and the sound event timestamp features are subsequently processed by the corresponding feature adapter to generate the corresponding second adjustment features and adjust the second output corresponding to each coding layer of the diffusion model during the diffusion processing, so as to inject the information of the timestamps corresponding to each sound event in the target video into the diffusion processing process, so that the accompanying audio finally generated by the diffusion model can be more accurately aligned with the target video in the time dimension.
[0267] In one embodiment, referring to Fig.23 , get the sound event timestamp features of the target video, also including:
[0268] Step 2310, performing sound event detection based on the visual features of the target video, determining at least one sound event in the target video, and determining at least one time node corresponding to the at least one sound event in the target video;
[0269] Step 2320, determining a timestamp feature of the sound event based on at least one time node.
[0270] In step 2310, the time node corresponding to the sound event in the target video refers to the time node corresponding to the moment when the sound event occurs in the target video. For example, if there is a picture of a puppy barking at the 5th second of the target video, then the 5th second of the target video is the time node corresponding to the sound event of the dog barking. By detecting the time nodes of each sound event in the target video, some short sound events in the target video can be accurately captured, such as the time nodes corresponding to the sound events such as hammering. In this way, the sound event timestamp feature is subsequently generated based on the detected time node, and the second output is adjusted according to the sound event timestamp feature, so that the sound effects corresponding to some short sound events in the generated accompanying audio can be accurately aligned with the picture corresponding to the sound event in the time dimension, avoiding the problem of audio and video being out of sync.
[0271] Specifically, refer to Fig.24, the object can choose to automatically obtain the timestamp in the interface of the audio generation software. After that, the visual features are called to detect sound events by calling a trained multimodal model for detecting sound events based on visual information to determine the time nodes of each sound event in the target video. For example, the target video is subjected to event detection to determine the events that occurred in the target video, and then the events detected from the target video are classified to determine whether the detected event is a sound event. For example, when a hammering event is detected, since the hammering event is often accompanied by the sound of hammering, the event can be determined as a sound event based on this, and the time when the event occurred in the target video is determined, thereby determining the time node corresponding to the sound event.
[0272] In another embodiment, the time node at which the sound event occurs in the target video may also be automatically detected by a timestamp detector, for example, by detecting the time node at which the sound event occurs in the sample video by using the librosa.onset.onset_detect module.
[0273] In step 2320, after performing sound event detection on the target video and determining at least one time node corresponding to at least one sound event contained in the target video, the information of these time nodes is encoded to obtain corresponding sound event timestamp features.
[0274] In the embodiment disclosed in steps 2310 to 2320, by performing sound event detection on the target video, the start time of each sound event is determined as the time node corresponding to the sound event. The information of these time nodes is then encoded into a sound event timestamp feature. In this way, the sound event timestamp feature can accurately characterize the time nodes of each sound event contained in the target video. Subsequently, a second adjustment feature is generated based on the sound event timestamp feature, and the second output is adjusted accordingly, thereby injecting the information of the sound event timestamp feature into the diffusion processing process, so that the generated accompanying audio can be more accurately aligned with the target video picture in the time dimension.
[0275] In one embodiment, referring to Fig.25 , step 1730 includes:
[0276] Step 2510, obtaining a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, wherein a sum of the first weight coefficient and the second weight coefficient is not greater than a first threshold;
[0277] Step 2520, weighting the first adjustment feature and the second adjustment feature based on the first weight coefficient and the second weight coefficient respectively to obtain the weighted first adjustment feature and the weighted second adjustment feature, and adding the weighted first adjustment feature and the weighted second adjustment feature to the output of the corresponding coding layer.
[0278] In step 2510, it can be understood that when the first output of the coding layer needs to be adjusted based on the target sound event semantic feature and the sound event timestamp feature at the same time, since the semantic feature adapter for generating the first adjustment feature and the timestamp feature adapter for generating the second adjustment feature are trained separately, the process of generating the first adjustment feature based on the target sound event semantic feature and the process of generating the second adjustment feature based on the sound event timestamp feature are independent of each other. The first adjustment feature and the second adjustment feature are both feature matrices with the same shape as the first output. At this time, the first adjustment feature and the second adjustment feature are simultaneously superimposed on the first output of the corresponding coding layer. The first output may be distorted due to the excessive sum of the adjustment amplitudes of the first adjustment feature and the second adjustment feature, resulting in a decrease in the quality of the accompanying audio generated in the end. Based on this, in this embodiment, the first weight coefficient corresponding to the first adjustment feature and the second weight coefficient corresponding to the second adjustment feature are obtained, and the sum of the first weight coefficient and the second weight coefficient is controlled not to be greater than a preset first threshold value, and the first threshold value can be 1, thereby controlling the sum of the adjustment amplitudes of the first adjustment feature and the second adjustment feature.
[0279] In step 2520, after determining the first weight coefficient and the second weight coefficient, the first adjustment feature is weighted based on the first weight coefficient, and the second adjustment feature is weighted based on the second weight coefficient to obtain the weighted first adjustment feature and the weighted second adjustment feature, and then the weighted first adjustment feature and the weighted second adjustment feature are superimposed on the first output of the corresponding coding layer. Based on this, it can be avoided that when the first output is adjusted by the first adjustment feature and the second adjustment feature at the same time, the first output is distorted due to the sum of the adjustment amplitudes of the first adjustment feature and the second adjustment feature being too large. In this way, when the target sound event semantic features and the sound event timestamp features are injected into the diffusion processing process at the same time, the audio generation capability of the diffusion model can be avoided to be affected by the information of the injected target sound event semantic features and the sound event timestamp features.
[0280] In the embodiment disclosed in steps 2510 to 2520, a first weight coefficient and a second weight coefficient are obtained, and the first adjustment and the second adjustment are weighted based on the first weight coefficient and the second weight coefficient, respectively, to obtain a weighted first adjustment feature and a weighted second adjustment feature, and then the weighted first adjustment feature and the weighted second adjustment feature are superimposed on the first output of the corresponding coding layer, so that when the diffusion processing process is adjusted based on the first adjustment feature and the second adjustment feature, the quality of the generated accompanying audio is avoided to be reduced due to the sum of the adjustment amplitudes being too large, so that the diffusion processing process can accurately generate accompanying audio that is accurately aligned with the target video in both time dimension and semantics based on the target sound event semantic features and the sound event timestamp features.
[0281] The training process of the semantic feature generation model:
[0282] Since the diffusion model itself is used to generate audio based on visual features, what the diffusion model has learned is how to understand the visual features of the video and generate audio based on them, but it has not learned how to generate audio based on text features. When encoding the semantic description of a sound event into the corresponding semantic features of the target sound event, it is necessary to make the target sound event semantic features and the corresponding video features have a high degree of similarity, so that when the first output of the encoding layer is subsequently adjusted based on the first adjustment feature, the information of the target sound event semantic features can be more accurately injected into the diffusion processing process, and the diffusion model can capture the information contained in the semantic features of the target sound event. Based on this, it is necessary to train the semantic feature generation model in advance so that the semantic feature generation model can encode the given semantic description of a sound event into text features that have a high degree of similarity with the video features of the video corresponding to the semantic description of the sound event. Specifically, referring to Fig.26 In one embodiment, the semantic feature generation model can be pre-trained in the following manner:
[0283] Step 2610, obtaining a first training sample set, wherein the first training sample set includes a plurality of video-text pairs, and a single video-text pair includes a sample video and a sample sound event semantic description that correspond to each other;
[0284] Step 2620, encoding each sample video through a video feature generation model, and encoding each sample sound event semantic description through a semantic feature generation model, to obtain a plurality of sample video features and sample sound event semantic description features;
[0285] Step 2630, predicting a first matching probability between each sample video feature and each sample sound event semantic description feature, and determining a second matching probability between each sample video feature and each sample sound event semantic description feature based on a corresponding relationship between the sample video and the sample sound event semantic description;
[0286] Step 2640, determine a first loss value based on the first matching probability and the second matching probability, and optimize the semantic feature generation model and the video feature generation model based on the first loss value until the first loss value is less than the first loss threshold, thereby obtaining a semantic feature generation model.
[0287] In step 2610, the first training sample set is a data set constructed by sample videos and sample sound event semantic descriptions. The sample videos and sample sound event semantic descriptions in the first training sample set are one-to-one corresponding. The sample sound event semantic description is a text used to describe the video content in the corresponding sample video. For example, the content of a sample video is a person hammering an object with a hammer, and the corresponding sample sound event semantic description is "hammering an object with a hammer."
[0288] In step 2620, the video feature generation model is a video modality encoder, which can extract multiple sample video frames from the sample video, then perform convolution processing on the sample video frames to obtain sample video frame features corresponding to each sample video frame, and fuse the sample video frame features through a fully connected layer to obtain sample video features. It should be noted that the video feature generation model in step 2320 is different from the visual encoder mentioned in the above step 310 for extracting visual features from the target video, and both are pre-trained in different ways.
[0289] The semantic feature generation model is a text modality encoder, which converts the semantic description of the sample sound event into multiple words, then encodes each word into a corresponding feature vector, and then concatenates the feature vectors corresponding to each word to obtain the corresponding sample sound event semantic description features. The sample sound event semantic description features can be used to characterize the semantic features of the sample sound event semantic description.
[0290] In step 2630, the first matching probability is a probability distribution obtained by predicting the matching probability between each sample sound event semantic description feature and the sample video feature. For example, the first training sample set includes 100 one-to-one corresponding sample videos and sample sound event semantic descriptions. After being processed in step 2620, 100 sample video features and sample sound event semantic description features are obtained; then, for each sample sound event semantic description feature, the matching probability between the sample sound event semantic description feature and the 100 sample video features is predicted; accordingly, for each sample video feature, the matching probability between the sample video feature and the 100 sample sound event semantic description features is predicted. Specifically, the cosine similarity score between the sample video feature and the sample sound event semantic description feature can be calculated, and then the similarity score between the single sample sound event semantic description feature and each sample video feature can be exponentially normalized by the softmax function. For example, when the first training sample set includes 100 one-to-one corresponding sample videos and sample sound event semantic descriptions, for each sample sound event semantic description feature, the cosine similarity between the sample sound event semantic description feature and the 100 sample video features is calculated to obtain the corresponding 100 cosine similarity scores, and then the 100 cosine similarity scores are exponentially normalized, thereby obtaining the matching probability distribution between the sample sound event semantic description feature and each sample video feature; similarly, the matching probability distribution between the sample video feature and each sample sound event semantic description feature can also be obtained. Thus, the first matching probability can be determined.
[0291] The second matching probability is a probability distribution used to characterize the true correspondence between the sample video and the sample sound event semantic description. It is not difficult to understand that in the second matching probability, the matching probability between each sample video and the corresponding sample sound event semantic description is 1, while the matching probability between the semantic descriptions of other sample sound events in the first training sample set is 0. For example, the first training sample set includes 100 one-to-one corresponding sample videos and sample sound event semantic descriptions, and the 10th sample video corresponds to the 10th sample sound event semantic description. Then, in the second matching probability, the matching probability between the sample video feature corresponding to the 10th sample video and the sample sound event semantic description feature corresponding to the 10th sample sound event semantic description is 1, and the matching probability between the sample video feature corresponding to the 10th sample video and the sample sound event semantic description feature of the other 99 sample sound event semantic descriptions in the first training sample set is 0, thereby determining the matching probability distribution corresponding to each sample video feature in the second matching probability; similarly, the matching probability distribution corresponding to each sample sound event semantic description feature in the second matching probability can be determined to determine the second matching probability.
[0292] In step 2640, after determining the first matching probability and the second matching probability, the cross entropy between the first matching probability and the second matching probability can be calculated as the first loss value. Cross entropy can be used to measure the similarity between two probability distributions. The more similar the two probability distributions are, the smaller the corresponding cross entropy will be. By using the cross entropy as the first loss value, the semantic feature generation model and the video feature generation model can learn how to encode text and video with similar semantics into similar feature representations. After the training is completed, the semantic feature generation model is used to encode the semantic description of the sound event, so that the target sound event semantic features obtained by encoding and the video features obtained by encoding the video used to represent the corresponding sound event have a high similarity. For example, if the semantic description of the sound event is "a person is hammering an object with a hammer", then the target sound event semantic features obtained by encoding the semantic description of the sound event by the trained semantic feature generation model and the video features obtained by encoding the video of "a person hammering an object with a hammer" using the video generation model have a high similarity.
[0293] In step 2610 to step 2640, by using the video feature generation model and the semantic feature generation model, the sample video and the sample sound event semantic description are respectively encoded into corresponding sample video features and sample sound event semantic description features, and then the first matching probability between the sample video and the sample sound event semantic description is predicted, and the second matching probability is determined based on the corresponding relationship between the sample video and the sample sound event semantic description, and the cross entropy between the first matching probability and the second matching probability is used as the first loss value to optimize the parameters of the video feature generation model and the semantic feature generation model, so that the video feature generation model and the semantic feature generation model can encode semantically similar videos and texts into video features and text features with high similarity. In this way, the semantic feature generation model after training is used to encode the semantic description of the sound event, and the obtained semantic features of the sound event and the video features obtained by encoding the video used to represent the corresponding sound event have a high similarity.
[0294] The training process of feature adapter:
[0295] Reference Fig. 27 In one embodiment, the semantic feature adapter can be trained by:
[0296] Step 2710, copying the coding layer in the diffusion model to obtain an initialized semantic feature adapter, and freezing the parameters of the diffusion model, the semantic feature adapter including adapter layers corresponding to the plurality of coding layers of the diffusion model;
[0297] Step 2720, obtaining a second training sample set, where the second training sample set includes at least one sample content;
[0298] Step 2730, extracting sample original audio features, sample target sound event semantic features and sample content features from the sample content;
[0299] Step 2740, performing multiple noise addition steps on the sample original audio features through the diffusion model to obtain a noisy audio feature, and storing the forward noise added to the sample original audio feature in each noise addition step;
[0300] Step 2750, inputting the sample content feature into the diffusion model, so that the diffusion model performs multiple denoising steps on the noise audio feature based on the sample content feature; in each denoising step, inputting the sample target sound event semantic feature into the semantic feature adapter, obtaining the sample adjustment feature corresponding to each coding layer, adjusting the output of the corresponding coding layer based on the sample adjustment feature, so as to obtain the reverse noise corresponding to the single denoising step, and the multiple denoising steps correspond to the multiple noise adding steps;
[0301] Step 2760, determine a second loss value based on the reverse noise and the forward noise of the corresponding noise addition step, and optimize the parameters of each adapter layer based on the second loss value until the semantic feature adapter meets the first condition, thereby obtaining a semantic feature adapter.
[0302] In step 2710, at the beginning of training, the initial semantic feature adapter can be obtained by copying the structure and parameters of each encoding layer of the diffusion model. Fig.28 The diffusion model is based on the Unet network. The diffusion model includes encoding layers A to D, decoding layers A to D corresponding to encoding layers A to D, and an intermediate connection layer, which can also be regarded as an encoding layer. After copying the parameters of the diffusion model, adapter layers A to D and the intermediate adapter layer are obtained, and the adapter layer constitutes a semantic feature adapter. After copying the parameters of the diffusion model and obtaining the initial semantic feature adapter, the parameters of the diffusion model itself are frozen to avoid changes in the parameters of the diffusion model itself during the training process. Since the semantic feature adapter is obtained by copying the parameters and structure of multiple encoding layers of the diffusion model itself, the structure of the neural network layer in the semantic feature adapter is consistent with the structure of multiple encoding layers in the diffusion model, and the semantic feature adapter will include adapter layers corresponding to each encoding layer. In addition, referring to Fig.28A zero convolution layer is also set between the semantic feature adapter and the diffusion model. This zero convolution layer is a 1*1 convolution layer. At the beginning of training, the weight and bias of the zero convolution layer are set to 0. In this way, the harmful noise generated by the semantic feature adapter can be avoided from affecting the output of each encoding layer of the diffusion model at the beginning of training. During the training process, the parameters of the adapter layer and the zero convolution layer will be optimized synchronously.
[0303] In step 2720, after obtaining the initial semantic feature adapter, a second training sample set for training the semantic feature adapter is obtained, the second training sample set includes a plurality of sample contents, and the sample content may be a sample audio video with sound effects. Exemplarily, the second training sample set may be composed of two data sets, AudioSet and Vggsound.
[0304] Since the sample content is often obtained from the Internet, and the audio quality of the audio videos on the Internet varies, it often carries a lot of noise, background sound and other interfering sound effects. If only the sample audio video is used as the sample content, this will cause the sample original audio features extracted from the sample content to carry a lot of noise and background sound. When training the semantic feature adapter, the semantic feature adapter needs to learn how to predict the corresponding noise based on the given sample target sound event semantic features and sample content features, and restore the sample original audio features from the noise audio features. Since the sample original audio features themselves carry interfering sound effects such as noise and background sound, this causes the semantic feature adapter to mistakenly learn the parts related to these noises and background sounds during the training process. Based on this, in one embodiment, in addition to the sample audio video, the sample audio can also be used as the sample content, so that more high-quality noise-free sample audio can be used as the sample content, reducing the noise and background sound components in the sample original audio features extracted from the sample content, and avoiding the semantic feature adapter from mistakenly learning knowledge related to noise and background sound.
[0305] In step 2730, the sample original audio feature is a feature representation corresponding to the audio contained in the sample content. When the sample content is a sample audio, the sample audio can be directly processed to obtain a corresponding feature representation; when the sample content is a sample audio video, the audio of the sample audio video can be extracted and processed to obtain a corresponding feature representation. In one embodiment, the sample audio or the audio in the sample audio video can be processed by Mel cepstrum to obtain a Mel cepstrum feature, which can be used as the corresponding sample original audio feature.
[0306] The method for obtaining the semantic features of the sample sound event can refer to the method for obtaining the semantic features of the target sound event corresponding to the target video in the above embodiment. Fig.29 , by extracting multiple video frames from the sample audio video and inputting them into the visual language model after adding corresponding preset prompts, the corresponding sample sound event semantic description can be generated, and then the sample sound event semantic description can be encoded into the corresponding sample sound event semantic feature through the trained semantic feature generation model; when the sample content is sample audio, the sample audio can be directly input into the corresponding audio language model to generate the corresponding sample audio description text.
[0307] The sample content features are used to input into the diffusion model to guide the diffusion process of the diffusion model. Fig.29 When the sample content is a sample audio and video, the sample audio and video is encoded by a visual encoder to obtain the corresponding sample content features; when the sample content is a sample audio, the sample audio is encoded by an audio encoder to obtain the sample audio features as the sample content features.
[0308] It should be noted that the sample visual features are obtained by encoding the sample sound video through the visual encoder, and the sample audio features are obtained by encoding the sample audio through the audio encoder, wherein the visual encoder and the audio encoder are jointly trained, and the two can encode the temporally and semantically aligned video and audio into visual features and audio features with high similarity and aligned in the temporal and semantic dimensions, respectively. The training process of the visual encoder and the audio encoder will be explained later and will not be repeated here.
[0309] In step 2740, the sample original audio features are first subjected to multiple noise addition steps by the diffusion model, thereby performing noise addition processing on the sample original audio features to obtain a noisy audio feature. At the same time, the noise added to the sample original audio features by each diffusion model in each noise addition step, that is, the forward noise corresponding to each noise addition step, is recorded. Specifically, refer to Fig.29 The original audio features of the sample are first encoded through a variational autoencoder to obtain the corresponding feature representation, and then the feature representation is input into the diffusion model. During the forward diffusion process, the diffusion model directly predicts noise based on the feature representation, generates noise corresponding to each noise addition step, and adds it to the feature representation, thereby gradually transforming the original audio features of the sample into noisy audio features.
[0310] In step 2750, the diffusion model is used to perform multiple denoising steps on the noisy audio features based on the sample content features to perform diffusion processing. The process of performing diffusion processing on the diffuse audio features based on the visual features in step 320 can be referred to. Fig.29For example, when the sample content is a sample audio video, multiple key frames can be extracted from the sample content and input into the visual language model to obtain the corresponding sample sound event semantic description. Then, the sample sound event semantic description is encoded through the semantic feature generation model to obtain the sample target sound event semantic feature.
[0311] In each denoising step of the diffusion process, the semantic features of the sample target sound event are input into the semantic feature adapter, referring to Fig.28 In the adapter, firstly, a zero convolution layer is used to perform zero convolution processing on the semantic features of the sample target sound event, and at the same time, the input of the first coding layer of multiple coding layers in the diffusion model, that is, the input of coding layer A, is obtained, and the input of coding layer A and the semantic features of the sample target sound event after zero convolution processing are superimposed to obtain sample superposition features, which are input to the first adapter layer in the semantic feature adapter, that is, adapter layer A, and the sample superposition features are processed accordingly in adapter layer A to obtain sample adjustment features corresponding to adapter layer A. At the same time, multiple adapter layers in the semantic feature adapter are cascaded in sequence, and the output of adapter layer A is also used as the input of adapter layer B. Adapter layer B processes based on the corresponding input to obtain the corresponding output, that is, the sample adjustment features corresponding to coding layer B, which is also the input of adaptation layer C. By analogy, the sample adjustment features corresponding to each coding layer are obtained. Then, the sample adjustment features are superimposed on the output of the corresponding coding layer.
[0312] It should be noted that in the diffusion model of the Unet architecture, the output of each encoding layer is also used as the input of the next encoding layer, and the sample adjustment features are only superimposed on the output from the encoding layer to the corresponding decoding layer, but not on the output from the encoding layer to the next encoding layer. Fig.28 In the example, the sample adjustment features generated by the adapter layer A are only superimposed on the output of the encoding layer A passed to the decoding layer A, but not on the output of the encoding layer A passed to the encoding layer B.
[0313] In one embodiment, referring to Fig.28 ,The output end of each adapter layer is also connected to a zero convolution layer. After obtaining the output of each adapter layer, the output end of each adapter layer is connected to a zero convolution layer. After the output of the adapter layer is processed by zero convolution through the zero convolution layer, the sample adjustment features from the adapter layer input to the diffusion model are obtained.
[0314] In step 2760, the first condition is a preset condition for determining whether the semantic feature adapter has been trained. For example, the loss value during the training process may be compared with a set loss threshold. Alternatively, the number of trainings may be set. For example, after training the semantic feature adapter with 300 sample contents, the obtained semantic feature adapter is tested using a test set. Based on the test set, the performance of the semantic feature adapter is evaluated to determine whether it meets the requirements, thereby determining whether the semantic feature adapter has been trained.
[0315] Specifically, since each denoising step corresponds to a noise adding step, by comparing the noise predicted by each denoising step with the noise added to the original audio features of the sample in the corresponding noise adding step, the corresponding loss value, i.e., the second loss value, can be calculated, and then back propagation is performed based on the second loss value. It should be noted that since the parameters of the diffusion model itself are frozen, in the process of back propagation, only the gradients corresponding to each adapter layer in the semantic feature adapter and the gradients of the zero convolution layer used to connect the semantic feature adapter and the diffusion model are calculated based on the chain rule, while the gradients of the encoding layer and other neural network layers of the diffusion model itself are not calculated. Afterwards, the zero convolution layer and each adapter layer perform gradient descent processing on the corresponding parameters based on the corresponding gradients, thereby optimizing the corresponding parameters until the semantic feature adapter meets the set first condition, which indicates that the semantic feature adapter has well learned how to generate the corresponding first adjustment features based on the input target sound event semantic features. At this time, a trained semantic feature adapter can be obtained.
[0316] In the embodiment disclosed in steps 2710 to 2760, the parameters of each coding layer of the diffusion model are copied to obtain an initialized semantic feature adapter, and the parameters of the diffusion model are frozen at the same time. Then, the corresponding sample content features, sample original audio features and sample target sound event semantic features are extracted from the sample content, and the sample original audio features are denoised to become noise audio features. Then, the sample content features are input into the diffusion model, so that the diffusion model performs multiple denoising steps on the noise audio features based on the sample content features. At the same time, in each denoising step, the sample target sound event semantic features are input into the semantic feature adapter, so that the sample target sound event semantic features are processed accordingly through the semantic feature adapter, and the sample adjustment features corresponding to each coding layer are obtained, so as to adjust the outputs of each coding layer of the diffusion model obtained during the diffusion processing process, thereby predicting the noise corresponding to the denoising step. Afterwards, the corresponding second loss value is calculated based on the noise predicted by the denoising step and the noise added to the original audio features of the sample in the denoising step corresponding to the denoising step, and then back propagation is performed based on the second loss value to calculate the gradients corresponding to each adapter layer, and the parameters of each adapter layer are optimized by gradient descent. The semantic feature adapter meets the first condition, and in this way, the semantic feature adapter learns how to process the semantic features of the target sound event into corresponding feature representations to be injected into the diffusion processing process. When the semantic feature adapter is trained, the semantic features of the target sound event can be injected into the diffusion processing process through the semantic feature adapter, so that the diffusion model can generate accompanying audio that is accurately aligned with the target video in the semantic dimension based on the additionally injected semantic features of the target sound event.
[0317] The training process of the semantic feature adapter is shown in the above steps 2710 to 2760. It can be understood that the training process of the timestamp feature adapter used to generate the second adjustment feature based on the sound event timestamp feature can also refer to the training process of training the semantic feature adapter in steps 2710 to 2760. The difference is that when training the timestamp feature adapter, the sample sound event timestamp feature is extracted from the sample content as the input of the timestamp feature adapter, without extracting the sample target sound event semantic feature from the sample content. For the sample sound event timestamp feature, since the sample sound video and the sample audio themselves carry audio signals, sound event detection can be performed on the audio signal carried by the sample content to determine the time node of the occurrence of the sound event as the corresponding sample sound event timestamp feature. For example, the audio event detection of the audio signal carried by the sample content is performed through the ATST-Frame model to determine the corresponding timestamp feature of the sound event; the corresponding timestamp feature of the sample sound event can also be determined by detecting the audio energy of the audio signal of the sample content, for example, the timestamp feature of the sample sound event in the sample audio or sample audio video is detected through the librosa.onset.onset_detect module in the librosa toolkit. After obtaining the timestamp feature of the sample sound event, the timestamp feature of the sample sound event is input into the timestamp feature adapter to determine the corresponding second adjustment feature of the sample. Based on the second adjustment feature of the sample, the output of the encoding layer of the diffusion model is adjusted so that the diffusion model predicts the corresponding noise based on the timestamp feature of the sample sound event, and the corresponding loss value is determined in combination with the noise added to the original audio feature of the sample in the corresponding noise addition step. Back propagation is performed according to the loss value to determine the gradient of the timestamp feature adapter, and the parameters of the timestamp feature adapter are optimized by gradient descent until the timestamp feature adapter meets the first condition, and a trained timestamp feature adapter is obtained.
[0318] Training process of visual encoder and audio encoder:
[0319] When using sample audio as sample content, the sample audio needs to be encoded through an audio encoder to obtain corresponding sample audio features, which are input into the diffusion model as sample content features. Since the diffusion model itself predicts the corresponding noise based on the visual features of the video, when using the sample audio as sample content to train the feature adapter, it is necessary to ensure that the sample audio features obtained based on the sample audio and the visual features obtained based on the video corresponding to the sample audio are aligned in the semantic and temporal dimensions. This requires joint training of the visual encoder used to encode the sample audio video into sample visual features and the audio encoder used to encode the sample audio into corresponding sample audio features.
[0320] Reference Fig.30 In one embodiment, the sample content is a sample audio or a sample video with sound, the sample content feature includes one of a sample visual feature and a sample audio feature, the sample visual feature is obtained by encoding the sample video with sound through a visual encoder, the sample audio feature is obtained by encoding the sample video with sound or the sample audio through an audio encoder, and the visual encoder and the audio encoder are pre-trained jointly in the following manner:
[0321] Step 3010, obtaining a third training sample set, wherein the third training sample set includes a plurality of sample audio and video pairs, and the sample audio and video pairs include corresponding training sample audio and training sample video;
[0322] Step 3020, encoding each training sample audio and each training sample video by an audio encoder and a visual encoder respectively, to obtain a plurality of training sample audio features and a plurality of training sample visual features;
[0323] Step 3030, dividing the training sample audio and the training sample video in the single sample audio-video pair into a plurality of sample audio segments and a plurality of sample video segments corresponding to each other, and encoding the plurality of sample audio segments and the plurality of sample video segments correspondingly through an audio encoder and a visual encoder, respectively, to obtain a plurality of sample segment audio features and a plurality of sample segment visual features;
[0324] Step 3040, determining a semantic alignment score based on a plurality of training sample audio features and a plurality of training sample visual features, and determining a temporal alignment score based on a plurality of sample segment audio features and a plurality of sample segment visual features;
[0325] Step 3050: determine an audio and video alignment score based on the semantic alignment score and the temporal alignment score, and optimize the audio encoder and the visual encoder based on the audio and video alignment score until the audio and video alignment score is greater than the alignment score threshold.
[0326] In step 3010, the third training sample set is a data set including multiple sample audio and video pairs, which includes multiple sample audio and video pairs, each sample audio and video pair includes a corresponding training sample video and a training sample audio, and the training sample video and training sample audio in a single sample audio and video pair are aligned in time and semantic dimensions.
[0327] In step 3020, each training sample audio is encoded by an audio encoder to obtain corresponding training sample audio features, and each training sample video is encoded by a visual encoder to obtain multiple training sample visual features. The sample visual features and sample audio features obtained based on the training sample video and the training sample audio in the same sample audio and video pair correspond to each other.
[0328] In step 3030, the training sample video and the training sample audio in each sample audio-video pair are respectively segmented into a plurality of sample audio segments and sample video segments. It is understandable that, when segmenting, the rules for segmenting the training sample video and the training sample audio are consistent. For example, when segmenting the training sample video, every 8 seconds in the training sample video is taken as a sample video segment. Then, when segmenting the training sample audio, every 8 seconds in the training sample audio is also segmented into a sample audio segment. Among them, the sample video segment consisting of the first 8 seconds of the training sample video corresponds to the sample audio segment consisting of the first 8 seconds of the training sample audio in the same sample audio-video pair. The sample video segment consisting of the 9th to 16th seconds of the training sample video corresponds to the sample audio segment consisting of the 9th to 16th seconds of the training sample audio in the same sample audio-video pair. And so on. This ensures that each sample video segment and sample audio segment obtained after segmentation can correspond one to one.
[0329] After the training sample video and the training sample audio are respectively divided into multiple sample video segments and sample audio segments, each sample video segment is encoded into a sample segment visual feature through a visual encoder, and each sample audio segment is encoded into a corresponding sample segment audio feature through an audio encoder.
[0330] In step 3040, after encoding the training sample video and the training sample audio in each sample audio-video pair respectively to obtain a plurality of training sample visual features and training sample audio features, the cosine similarity between each training sample visual feature and each training sample audio feature can be calculated to determine the matching score between the training sample audio feature and each training sample visual feature. Specifically, the matching score between the i-th training sample audio and the j-th training sample video can be expressed as:
[0331] ;
[0332] in, represents the matching score between the i-th training sample audio and the j-th training sample video, represents the number of sample audio and video pairs in the third training sample set, sim() represents the cosine similarity function, is a preset hyperparameter, represents the training sample audio feature corresponding to the i-th training sample audio, represents the visual features of the training sample corresponding to the jth training sample video, represents the visual features of the training sample corresponding to the kth training sample video, represents the training sample audio feature corresponding to the kth training sample audio. Then, the matching scores between each training sample video and each training sample audio are weighted based on the true matching probability between each training sample video and training sample audio to obtain the semantic alignment score. For example, the i-th training sample video and the i-th training sample audio correspond to each other, so The score is multiplied by the weight coefficient 1, and Multiply by 0. This gives us the semantic alignment score, which is recorded as .
[0333] After determining the semantic alignment score, because the audio and video need to be aligned in both semantics and time, the time alignment score can be calculated based on the segmentation of a single sample audio and video pair to obtain multiple sample video segments and sample audio segments. Specifically, the cosine similarity between the visual features of the sample segments and the audio features of the sample segments can be calculated to determine the matching scores between each sample video segment and the sample audio segment in a single sample audio and video pair. Specifically, the matching scores between the sample video segments and the sample audio segments can be determined by the following formula:
[0334] ;
[0335] in, represents the matching score between the i-th sample audio segment and the j-th sample video segment, represents the number of sample video segments or sample audio segments obtained from a single sample audio and video pair, sim() represents the cosine similarity function, is a preset hyperparameter, represents the audio features of the sample segment corresponding to the i-th sample audio segment, represents the visual features of the sample segment corresponding to the jth sample video segment, represents the visual features of the sample segment corresponding to the k-th sample video segment, represents the audio features of the sample segment corresponding to the kth sample audio segment. Based on the true correspondence between each sample video segment and each sample audio segment corresponding to a single sample audio and video pair, the matching scores between each sample video segment and each sample audio segment are weighted to obtain a time alignment score. For example, the i-th sample video segment corresponds to the i-th sample audio segment. Therefore, The score is multiplied by the weight coefficient 1, and Multiply by 0. This gives the time alignment score, which is recorded as .
[0336] In step 3050, the audio and video alignment score is the sum of the scores obtained by weighted summation of the time alignment score and the semantic alignment score; the alignment score threshold is a preset threshold. Specifically, after determining the time alignment score and the semantic alignment score, the audio and video alignment score can be determined by the following formula: + ,in, represents the audio and video alignment score, λ is a preset coefficient. In this embodiment, The value is 1. The audio-video alignment score characterizes the alignment of the audio features of the training samples and the audio features of the sample segments obtained by the audio encoder with the visual features of the training samples and the visual features of the sample segments obtained by the visual encoder in the temporal dimension and the semantic dimension. By comparing the audio-video alignment score and the alignment score threshold, the audio encoder and the visual encoder are optimized to maximize the audio-video alignment score. When the audio-video alignment score is greater than the alignment score threshold, it indicates that the audio encoder and the visual encoder can encode the audio and video aligned in the temporal dimension and the semantic dimension into audio features and visual features with high similarity, respectively.
[0337] In the embodiment disclosed in steps 3010 to 3050, the visual encoder and the audio encoder are trained by multiple sample audio and video pairs, so that the audio encoder and the visual encoder can encode the audio and video aligned in the time dimension and the semantic dimension into audio features and visual features with high similarity, respectively. In this way, the sample audio can be used as the sample content, and the sample audio can be encoded into corresponding sample audio features by the audio encoder to train the feature adapter, so as to train the feature adapter with more high-quality audio without noise and background sound, so that the feature adapter can learn how to accurately encode the target sound event semantic features or sound event timestamp features into the corresponding first adjustment features or second adjustment features, so as to accurately adjust the first output of each coding layer of the diffusion model to generate accompanying audio that is semantically and temporally aligned with the target video.
[0338] An example of a specific use process of the accompanying audio generation method of the disclosed embodiment:
[0339] Refer to the following Fig.31 , a specific use process of the accompanying audio generation method of the embodiment of the present disclosure is described in detail, and the process includes but is not limited to the following steps 3101 to 3122. In this process, the server 140 and the object terminal 110 each undertake a part of the task of the accompanying audio generation method. In actual use, the server 140 may also undertake all the tasks of the accompanying audio generation method.
[0340] 3101. The server 140 extracts visual features of the target video;
[0341] 3102. The server 140 decomposes the target video into video frames;
[0342] 3103. The server 140 inputs the decomposed video frames and the preset prompts into the visual language model, so that the visual language model generates a semantic description of the sound event based on the video frames;
[0343] 3104. The server 140 encodes the semantic description of the sound event through the semantic feature generation model to obtain the semantic features of the target sound event;
[0344] 3105. The target terminal 110 obtains the forward prompt word and the reverse prompt word, and sends the forward prompt word and the reverse prompt word to the server 140;
[0345] 3106. The server 140 encodes the forward prompt word and the reverse prompt word received from the target terminal 110 into corresponding prompt word features, and splices the prompt word features into the target sound event semantic features to update the target sound event semantic features;
[0346] 3107. The target terminal 110 obtains the first number and sends the first number to the server 140;
[0347] 3108. The server 140 generates a first number of random seeds based on the first number received from the target terminal 110, and generates a first number of audio features to be diffused based on the first number of random seeds;
[0348] 3109. The server 140 inputs the first number of audio features to be diffused into the diffusion model respectively, and performs diffusion processing on the first number of audio features to be diffused respectively based on the visual features;
[0349] 3110. The server 140 obtains the initial input of the first coding layer of the diffusion model;
[0350] 3111. The server 140 performs zero convolution processing on the semantic features of the target sound event, and superimposes the semantic features of the target sound event after the zero convolution processing and the initial input to obtain the superimposed semantic features of the target sound event;
[0351] 3112. The server 140 inputs the superimposed semantic features of the target sound event into the corresponding semantic feature adapter, and performs zero convolution processing on the output of the semantic feature adapter to obtain a first adjusted feature;
[0352] 3113. The target terminal 110 obtains a first operation on the target video, and sends the first operation to the server 140;
[0353] 3114. The server 140 determines a plurality of first annotation information based on the first operation received from the object terminal 110, and determines a sound event timestamp feature based on the plurality of first annotation information;
[0354] 3115. The server 140 performs sound event detection based on the visual features of the target video, determines at least one sound event in the target video, and determines at least one time node corresponding to the at least one sound event in the target video;
[0355] 3116. The server 140 determines a timestamp feature of the sound event based on at least one time node;
[0356] 3117. The server 140 generates a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature;
[0357] 3118. The server 140 determines the intermediate feature of the audio to be diffused based on the feature of the audio to be diffused in the encoding layer;
[0358] 3119. The server 140 generates a query matrix based on the intermediate features of the audio to be diffused, and generates a key matrix and a value matrix based on the visual features;
[0359] 3120. The server 140 performs cross attention processing based on the query matrix, the key matrix, and the value matrix, and determines a first output corresponding to the encoding layer based on a result of the cross attention processing;
[0360] 3121. The target terminal 110 obtains a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, and sends the first weight coefficient and the second weight coefficient to the server 140;
[0361] 3122. The server 140 performs weighted processing on the first adjustment feature and the second adjustment feature based on the first weight coefficient and the second weight coefficient received from the target terminal 110, respectively, to obtain the weighted first adjustment feature and the weighted second adjustment feature, and accumulates the weighted first adjustment feature and the weighted second adjustment feature to the first output of the corresponding coding layer to obtain a third output;
[0362] 3123. The server 140 generates accompanying audio for the target video based on the third output, so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event according to the timestamp corresponding to the timestamp feature of the sound event.
[0363] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.
[0364] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present application is obtained.
[0365] Description of the apparatus and device of the embodiments of the present disclosure:
[0366] Reference Fig.32 , Fig.32 The accompanying audio generating device 3200 provided in an embodiment of the present disclosure is a schematic diagram of the structure of the accompanying audio generating device 3200, and the accompanying audio generating device 3200 includes:
[0367] A feature extraction unit 3210 is used to extract visual features of a target video and obtain semantic features of a target sound event of the target video;
[0368] A diffusion execution unit 3220, configured to input the visual features into a diffusion model, the diffusion model comprising a coding layer for performing diffusion processing on the diffuse audio features according to the visual features;
[0369] The diffusion adjustment unit 3230 is used to generate a first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event, and adjust the first output of the coding layer based on the first adjustment feature to obtain a second output, and generate accompanying audio of the target video based on the second output so that the accompanying audio contains an audio clip corresponding to the semantic feature of the target sound event.
[0370] Optionally, the feature extraction unit 3210 is used to:
[0371] Identify sound events from a target video and obtain target frames containing the sound events;
[0372] Inputting the target frame and the context frame of the target frame in the target video into the sound event importance prediction model to obtain the predicted importance of the sound event;
[0373] Get the sound intensity corresponding to the sound event;
[0374] determining a target sound event among the identified sound events based on the importance and the sound intensity;
[0375] Obtain target sound event semantic features of the target sound event.
[0376] Optionally, the feature extraction unit 3210 is used to:
[0377] Decompose the target video into video frames;
[0378] The decomposed video frames and preset prompts are input into a visual language model, so that the visual language model generates a semantic description of the sound event based on the video frames, wherein the preset prompts are used to prompt the visual language model to generate a semantic description of the sound event based on the video frames;
[0379] The semantic description of the sound event is encoded through the semantic feature generation model to obtain the semantic features of the target sound event.
[0380] Optionally, the accompanying audio generating device 3200 further includes a first training unit (not shown), and the first training unit is used to:
[0381] Acquire a first training sample set, wherein the first training sample set includes a plurality of video-text pairs, and a single video-text pair includes a sample video and a sample sound event semantic description corresponding to each other;
[0382] Encoding each sample video through a video feature generation model, and encoding each sample sound event semantic description through a semantic feature generation model, to obtain multiple sample video features and sample sound event semantic description features;
[0383] Predicting a first matching probability between each sample video feature and each sample sound event semantic description feature, and determining a second matching probability between each sample video feature and each sample sound event semantic description feature based on a corresponding relationship between the sample video and the sample sound event semantic description;
[0384] A first loss value is determined based on the first matching probability and the second matching probability, and a semantic feature generation model and a video feature generation model are optimized based on the first loss value until the first loss value is less than a first loss threshold, thereby obtaining a semantic feature generation model.
[0385] Optionally, the feature extraction unit 3210 is used to:
[0386] Obtaining a positive prompt word and a reverse prompt word, and encoding the positive prompt word and the reverse prompt word into corresponding prompt word features; wherein the positive prompt word is used to prompt the sound event included in the generated accompanying audio by the diffusion model, and the reverse prompt word is used to prompt the sound event not included in the generated accompanying audio by the diffusion model;
[0387] The prompt word features are concatenated to the target sound event semantic features to update the target sound event semantic features.
[0388] Optionally, the diffusion execution unit 3220 is used to:
[0389] In the encoding layer, determining the intermediate features of the audio to be diffused based on the features of the audio to be diffused;
[0390] Generate a query matrix based on the intermediate features of the audio to be diffused, and generate a key matrix and a value matrix based on the visual features;
[0391] A cross-attention process is performed based on the query matrix, the key matrix, and the value matrix, and a first output corresponding to the encoding layer is determined based on a result of the cross-attention process.
[0392] Optionally, the diffusion execution unit 3220 is used to:
[0393] Obtaining a first number, generating a first number of random seeds, and generating a first number of audio features to be diffused based on the first number of random seeds;
[0394] Inputting a first number of audio features to be diffused into a diffusion model respectively, and performing diffusion processing on the first number of audio features to be diffused based on visual features to obtain a first number of candidate accompanying audios;
[0395] Accompanying audio is determined from a first number of candidate accompanying audios.
[0396] Optionally, the diffusion adjustment unit 3230 is used to:
[0397] Get the initial input to the first encoding layer of the diffusion model;
[0398] Performing zero convolution processing on the semantic features of the target sound event, and superimposing the semantic features of the target sound event after the zero convolution processing and the initial input to obtain the superimposed semantic features of the target sound event;
[0399] The superimposed semantic features of the target sound event are input into the corresponding semantic feature adapter, and the output of the semantic feature adapter is subjected to zero convolution processing to obtain the first adjustment feature.
[0400] Optionally, the accompanying audio generating device 3200 further includes a second training unit (not shown), and the second training unit is used for:
[0401] Copying the encoding layer in the diffusion model to obtain an initialized semantic feature adapter, and freezing the parameters of the diffusion model, the semantic feature adapter includes adapter layers corresponding to the multiple encoding layers of the diffusion model;
[0402] Acquire a second training sample set, where the second training sample set includes at least one sample content;
[0403] Extracting sample original audio features, sample target sound event semantic features and sample content features from the sample content;
[0404] Perform multiple noise addition steps on the original audio features of the sample through the diffusion model to obtain the noisy audio features, and store the forward noise added to the original audio features of the sample in each noise addition step;
[0405] The sample content feature is input into the diffusion model so that the diffusion model performs multiple denoising steps on the noise audio feature based on the sample content feature; in each denoising step, the sample target sound event semantic feature is input into the semantic feature adapter to obtain the sample adjustment feature corresponding to each coding layer, and the output of the corresponding coding layer is adjusted based on the sample adjustment feature to obtain the reverse noise corresponding to the single denoising step, and the multiple denoising steps correspond to the multiple noise adding steps;
[0406] A second loss value is determined based on the reverse noise and the forward noise of the corresponding noise adding step, and the parameters of each adapter layer are optimized based on the second loss value until the semantic feature adapter meets the first condition, thereby obtaining a semantic feature adapter.
[0407] Optionally, the accompanying audio generating device 3200 further includes a third training unit (not shown), and the third training unit is used to:
[0408] Acquire a third training sample set, wherein the third training sample set includes a plurality of sample audio and video pairs, and the sample audio and video pairs include corresponding training sample audio and training sample video;
[0409] Encode each training sample audio and each training sample video respectively through an audio encoder and a visual encoder to obtain multiple training sample audio features and multiple training sample visual features;
[0410] The training sample audio and the training sample video in a single sample audio-video pair are respectively divided into a plurality of sample audio segments and a plurality of sample video segments corresponding to each other, and the plurality of sample audio segments and the plurality of sample video segments are respectively encoded by an audio encoder and a visual encoder to obtain a plurality of sample segment audio features and a plurality of sample segment visual features;
[0411] Determine a semantic alignment score based on multiple training sample audio features and multiple training sample visual features, and determine a temporal alignment score based on multiple sample segment audio features and multiple sample segment visual features;
[0412] An audio and video alignment score is determined based on the semantic alignment score and the temporal alignment score, and an audio encoder and a visual encoder are optimized based on the audio and video alignment score until the audio and video alignment score is greater than an alignment score threshold.
[0413] Optionally, the feature extraction unit 3210 is used to:
[0414] Get the sound event timestamp features of the target video;
[0415] The diffusion adjustment unit 3230 is used for:
[0416] generating a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature;
[0417] adjusting the second output based on the second adjustment feature to obtain a third output;
[0418] Based on the third output, the accompanying audio of the target video is generated so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event according to the timestamp corresponding to the timestamp feature of the sound event.
[0419] Optionally, the feature extraction unit 3210 is used to:
[0420] From the target video, identify the target sound event;
[0421] Get the duration of the target sound event;
[0422] Get the event type of the target sound event;
[0423] If the duration and event type meet the predetermined conditions, the sound event timestamp feature of the target video is obtained.
[0424] Optionally, the feature extraction unit 3210 is used to:
[0425] If the target sound event is a subject speech event, identifying the subject's mouth size in a frame in the target video corresponding to the subject speech event;
[0426] If the size of the subject's mouth exceeds a predetermined size, a sound event timestamp feature of the target video is obtained.
[0427] Optionally, the feature extraction unit 3210 is used to:
[0428] In response to a first operation of the target terminal on the target video, determining a plurality of first annotation information, wherein the first annotation information is used to mark a time node at which a sound event occurs in the target video;
[0429] A sound event timestamp feature is determined based on the plurality of first annotation information.
[0430] Optionally, the feature extraction unit 3210 is used to:
[0431] Performing sound event detection based on visual features of the target video to determine at least one sound event in the target video, so as to determine at least one time node corresponding to the at least one sound event in the target video;
[0432] A sound event timestamp feature is determined based on at least one time node.
[0433] Optionally, the diffusion adjustment unit 3230 is used to:
[0434] Acquire a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, wherein a sum of the first weight coefficient and the second weight coefficient is not greater than a first threshold;
[0435] The first adjustment feature and the second adjustment feature are weighted based on the first weight coefficient and the second weight coefficient respectively to obtain the weighted first adjustment feature and the weighted second adjustment feature, and the weighted first adjustment feature and the weighted second adjustment feature are accumulated to the first output of the corresponding coding layer to obtain the third output.
[0436] Reference Fig.33 , Fig.33 The structural block diagram of the terminal part for implementing the accompanying audio generation method of the embodiment of the present disclosure includes: a radio frequency (RF) circuit 3310, a memory 3315, an input unit 3330, a display unit 3340, a sensor 3350, an audio circuit 3360, a wireless fidelity (WiFi) module 3370, a processor 3380, and a power supply 3390. Those skilled in the art can understand that Fig.33 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0437] The RF circuit 3310 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 3380 for processing; in addition, the designed uplink data is sent to the base station.
[0438] The memory 3315 may be used to store software programs and modules. The processor 3380 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 3315 .
[0439] The input unit 3330 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 3330 may include a touch panel 3331 and other input devices 3332.
[0440] The display unit 3340 may be used to display input information or provided information and various menus of the content terminal. The display unit 3340 may include a display panel 3341.
[0441] The audio circuit 3360, the speaker 3361, and the microphone 3362 may provide an audio interface.
[0442] In this embodiment, the processor 3380 included in the terminal can execute the accompanying audio generation method of the previous embodiment.
[0443] The terminals of the embodiments of the present disclosure include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to automatic picture matching, Wensheng video dubbing, etc.
[0444] Fig.34 A block diagram of the structure of a portion of a server 140 for implementing the accompanying audio generation method of an embodiment of the present disclosure. The server 140 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 3422 (for example, one or more processors) and memories 3432, and one or more storage media 3430 (for example, one or more mass storage devices) storing application programs 3442 or data 3444. Among them, the memories 3432 and the storage media 3430 may be temporary storage or permanent storage. The program stored in the storage medium 3430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server 140. Furthermore, the central processing unit 3422 may be configured to communicate with the storage medium 3430 to execute a series of instruction operations in the storage medium 3430 on the server 140.
[0445] The server 140 may also include one or more power supplies 3426, one or more wired or wireless network interfaces 3450, one or more input and output interfaces 3458, and / or one or more operating systems 3441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0446] The central processor 3422 in the server 140 can be used to execute the accompanying audio generation method of the embodiment of the present disclosure.
[0447] The embodiments of the present disclosure also provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the accompanying audio generation methods of the aforementioned embodiments.
[0448] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned accompanying audio generation method.
[0449] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar contents, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0450] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated content before and after is in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0451] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.
[0452] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0453] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0454] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0455] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server 140, or network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0456] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.
[0457] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.
Claims
1. A method for generating accompanying audio, characterized in that: The method comprises: Extracting visual features of a target video and obtaining semantic features of a target sound event of the target video; inputting the visual features into a diffusion model, the diffusion model comprising an encoding layer for diffusively processing diffuse audio features according to the visual features; Generate a first adjustment feature corresponding to the coding layer based on the target sound event semantic feature, adjust a first output of the coding layer based on the first adjustment feature to obtain a second output, and generate an accompanying audio of the target video based on the second output so that the accompanying audio contains an audio segment corresponding to the target sound event semantic feature; Among them, the method of generating the first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event includes: obtaining the initial input of the first coding layer of the diffusion model; performing zero convolution processing on the semantic feature of the target sound event, and superimposing the semantic feature of the target sound event after zero convolution processing and the initial input to obtain the superimposed semantic feature of the target sound event; inputting the superimposed semantic feature of the target sound event into the corresponding semantic feature adapter, and performing zero convolution processing on the output of the semantic feature adapter to obtain the first adjustment feature.
2. The accompanying audio generation method according to claim 1, characterized in that: The step of obtaining the semantic features of the target sound event of the target video includes: Identifying a sound event from the target video and acquiring a target frame containing the sound event; Inputting the target frame and the context frame of the target frame in the target video into a sound event importance prediction model to obtain a predicted importance of the sound event; Obtaining the sound intensity corresponding to the sound event; Determining a target sound event among the identified sound events based on the importance and the sound intensity; Acquire the target sound event semantic feature of the target sound event.
3. The accompanying audio generation method according to claim 1, characterized in that: The step of obtaining the semantic features of the target sound event of the target video includes: Decomposing the target video into video frames; Inputting the decomposed video frames and preset prompts into a visual language model, so that the visual language model generates a semantic description of the sound event based on the video frames, wherein the preset prompts are used to prompt the visual language model to generate the semantic description of the sound event based on the video frames; The semantic description of the sound event is encoded through a semantic feature generation model to obtain the semantic features of the target sound event.
4. The accompanying audio generation method according to claim 3, characterized in that: The semantic feature generation model is pre-trained in the following way: Acquire a first training sample set, wherein the first training sample set includes a plurality of video-text pairs, and a single video-text pair includes a sample video and a sample sound event semantic description corresponding to each other; Encoding each of the sample videos through the video feature generation model, and encoding each of the sample sound event semantic descriptions through the semantic feature generation model, to obtain a plurality of sample video features and sample sound event semantic description features; Predicting a first matching probability between each of the sample video features and each of the sample sound event semantic description features, and determining a second matching probability between each of the sample video features and each of the sample sound event semantic description features based on a corresponding relationship between the sample video and the sample sound event semantic description; A first loss value is determined based on the first matching probability and the second matching probability, and the semantic feature generation model and the video feature generation model are optimized based on the first loss value until the first loss value is less than a first loss threshold, thereby obtaining the semantic feature generation model.
5. The accompanying audio generation method according to claim 3, characterized in that: After encoding the semantic description of the sound event through the semantic feature generation model to obtain the semantic feature of the target sound event, the method further includes: Acquire a positive prompt word and a reverse prompt word, and encode the positive prompt word and the reverse prompt word into corresponding prompt word features; wherein the positive prompt word is used to prompt the diffusion model to generate a first reference sound event that the accompanying audio needs to correspond to, and the reverse prompt word is used to prompt the diffusion model to generate a second reference sound event that the accompanying audio cannot correspond to; The prompt word feature is spliced to the target sound event semantic feature to update the target sound event semantic feature.
6. The accompanying audio generation method according to claim 1, characterized in that: The step of performing diffusion processing on the diffuse audio feature according to the visual feature comprises: In the encoding layer, determining an intermediate feature of the audio to be diffused based on the feature of the audio to be diffused; Generate a query matrix based on the intermediate features of the audio to be diffused, and generate a key matrix and a value matrix based on the visual features; Cross-attention processing is performed based on the query matrix, the key matrix and the value matrix, and the first output corresponding to the encoding layer is determined based on the result of the cross-attention processing.
7. The accompanying audio generation method according to claim 1, characterized in that: The step of performing diffusion processing on the diffuse audio feature according to the visual feature further includes: Acquire a first number, generate a first number of random seeds, and generate a first number of the audio features to be diffused based on the first number of random seeds; Inputting the first number of audio features to be diffused into a diffusion model respectively, and performing diffusion processing on the first number of audio features to be diffused based on visual features respectively, so as to obtain a first number of candidate accompanying audios; Accompanying audio is determined from among the first number of candidate accompanying audios.
8. The accompanying audio generation method according to claim 1, characterized in that: The training process of the semantic feature adapter includes: Copying the coding layer in the diffusion model to obtain the initialized semantic feature adapter, and freezing the parameters of the diffusion model, the semantic feature adapter includes adapter layers corresponding to the plurality of coding layers of the diffusion model respectively; Acquire a second training sample set, where the second training sample set includes at least one sample content; Extracting sample original audio features, sample target sound event semantic features and sample content features from the sample content; Performing multiple noise addition steps on the sample original audio feature through the diffusion model to obtain a noisy audio feature, and storing the forward noise added to the sample original audio feature in each noise addition step; The sample content feature is input into a diffusion model so that the diffusion model performs multiple denoising steps on the noise audio feature based on the sample content feature; in each denoising step, the sample target sound event semantic feature is input into a semantic feature adapter to obtain a sample adjustment feature corresponding to each coding layer, and the output of the corresponding coding layer is adjusted based on the sample adjustment feature to obtain the reverse noise corresponding to a single denoising step, and multiple denoising steps correspond to multiple noise addition steps; A second loss value is determined based on the reverse noise and the corresponding forward noise of the noise adding step, and the parameters of each adapter layer are optimized based on the second loss value until the semantic feature adapter meets the first condition, thereby obtaining the semantic feature adapter.
9. The accompanying audio generation method according to claim 8, characterized in that: in, The sample content is a sample audio or a sample video with sound, the sample content feature includes one of a sample visual feature and a sample audio feature, the sample visual feature is obtained by encoding the sample video with sound through a visual encoder, the sample audio feature is obtained by encoding the sample video with sound or the sample audio through an audio encoder, and the visual encoder and the audio encoder are pre-jointly trained in the following manner: Acquire a third training sample set, wherein the third training sample set includes a plurality of sample audio and video pairs, and the sample audio and video pairs include corresponding training sample audio and training sample video; Encoding each of the training sample audios and each of the training sample videos respectively by the audio encoder and the visual encoder to obtain a plurality of training sample audio features and a plurality of training sample visual features; The training sample audio and the training sample video in the single sample audio-video pair are respectively divided into a plurality of sample audio segments and a plurality of sample video segments corresponding to each other, and the plurality of sample audio segments and the plurality of sample video segments are respectively encoded by the audio encoder and the visual encoder to obtain a plurality of sample segment audio features and a plurality of sample segment visual features; Determine a semantic alignment score based on the multiple training sample audio features and the multiple training sample visual features, and determine a temporal alignment score based on the multiple sample segment audio features and the multiple sample segment visual features; An audio-video alignment score is determined based on the semantic alignment score and the temporal alignment score, and the audio encoder and the visual encoder are optimized based on the audio-video alignment score until the audio-video alignment score is greater than an alignment score threshold.
10. The accompanying audio generation method according to claim 1, characterized in that: After acquiring the target sound event semantic features of the target video, the method further comprises: acquiring the sound event timestamp features of the target video; The step of generating the accompanying audio of the target video based on the second output so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event includes: generating a second adjustment feature corresponding to the coding layer based on the sound event timestamp feature; adjusting the second output based on the second adjustment feature to obtain a third output; The accompanying audio of the target video is generated based on the third output, so that the accompanying audio contains an audio segment corresponding to the semantic feature of the target sound event according to the timestamp corresponding to the timestamp feature of the sound event.
11. The accompanying audio generation method according to claim 10, characterized in that: The step of obtaining the timestamp feature of the sound event of the target video includes: From the target video, identifying a target sound event; Obtaining the duration of the target sound event; Obtaining the event type of the target sound event; If the duration and the event type meet a predetermined condition, a timestamp feature of the sound event of the target video is obtained.
12. The accompanying audio generation method according to claim 10, characterized in that: After identifying a target sound event from the target video, the method further includes: If the target sound event is a subject voice event, identifying a subject mouth size in a frame in the target video corresponding to the subject voice event; If the size of the object's mouth exceeds a predetermined size, a sound event timestamp feature of the target video is obtained.
13. The accompanying audio generation method according to claim 10, characterized in that: The step of obtaining the timestamp feature of the sound event of the target video includes: In response to a first operation of the target terminal on the target video, determining a plurality of first annotation information, wherein the first annotation information is used to mark a time node when a sound event occurs in the target video; The sound event timestamp feature is determined based on the plurality of first annotation information.
14. The accompanying audio generation method according to claim 10, characterized in that: The step of obtaining the timestamp feature of the sound event of the target video includes: Performing sound event detection based on visual features of the target video to determine at least one sound event in the target video, so as to determine at least one time node corresponding to the at least one sound event in the target video; The sound event timestamp feature is determined based on the at least one time node.
15. The accompanying audio generation method according to claim 10, characterized in that: The step of adjusting the second output based on the second adjustment feature to obtain a third output comprises: Acquire a first weight coefficient corresponding to the first adjustment feature and a second weight coefficient corresponding to the second adjustment feature, wherein a sum of the first weight coefficient and the second weight coefficient is not greater than a first threshold; The first adjustment feature and the second adjustment feature are weighted based on the first weight coefficient and the second weight coefficient respectively to obtain a weighted first adjustment feature and a weighted second adjustment feature, and the weighted first adjustment feature and the weighted second adjustment feature are accumulated to the first output of the corresponding coding layer to obtain the third output.
16. An accompanying audio generating device, characterized in that: include: A feature extraction unit, used to extract visual features of a target video and obtain semantic features of a target sound event of the target video; a diffusion execution unit, configured to input the visual features into a diffusion model, the diffusion model comprising a coding layer for performing diffusion processing on diffuse audio features according to the visual features; a diffusion adjustment unit, configured to generate a first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event, adjust a first output of the coding layer based on the first adjustment feature to obtain a second output, and generate an accompanying audio of the target video based on the second output, so that the accompanying audio includes an audio segment corresponding to the semantic feature of the target sound event; Among them, the method of generating the first adjustment feature corresponding to the coding layer based on the semantic feature of the target sound event includes: obtaining the initial input of the first coding layer of the diffusion model; performing zero convolution processing on the semantic feature of the target sound event, and superimposing the semantic feature of the target sound event after zero convolution processing and the initial input to obtain the superimposed semantic feature of the target sound event; inputting the superimposed semantic feature of the target sound event into the corresponding semantic feature adapter, and performing zero convolution processing on the output of the semantic feature adapter to obtain the first adjustment feature.
17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the accompanying audio generating method according to any one of claims 1 to 15 is implemented.
18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the accompanying audio generating method according to any one of claims 1 to 15 is implemented.
19. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the accompanying audio generation method according to any one of claims 1 to 15.
